A smart after-sales customer service method, system and storage medium

Abstract

This invention relates to the field of artificial intelligence technology, and discloses an intelligent after-sales customer service method, system, and storage medium. The intelligent after-sales customer service method includes the following steps: Step S101, determining the operation time and operation type; Step S102, determining the upper and lower bounds of the time window; Step S103, calculating the time prior density and time prior mask; Step S104, determining the event time; Step S105, outputting the answer probability distribution; Step S106, when the answer confidence is lower than the lower limit of the answer probability or the observation duration is less than the upper bound of the time window, outputting the additional shooting duration; otherwise, outputting the customer service answer. This invention accurately models the response delay after device operation, predicts the delay distribution based on device metadata and operation type, and determines the time window. It injects the time prior density and mask into an attention mechanism to constrain evidence retrieval to the effective time period, avoiding irrelevant interference and evidence collection during erroneous time periods.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and more specifically, to an intelligent after-sales customer service method, system, and storage medium. Background Technology

[0002] Currently, after-sales service relies heavily on short videos and text descriptions of devices uploaded by users. Mainstream algorithms often use methods such as uniform sampling of the entire video, average attention distribution, or fixed-length segment classification and matching, while also combining process documents to conduct text retrieval. These methods often assume that user operations and visible results on the device are approximately synchronized, or only use empirical time slices for rough positioning. They lack time-leading descriptions that adapt to different devices and different operations, and cannot finely constrain the time period when evidence appears.

[0003] Various devices are affected by firmware policies and physical processes, resulting in a common response delay. For example, after a network device restarts, it takes a certain amount of time for indicator lights and functions to recover. Air conditioners have a delayed start-up setting to avoid short cycles. Washing machine door locks and control boards have a waiting process for reset. Screens and indicator lights go through a transition phase after power failure. Furthermore, users often end their video recordings immediately after the operation or try the same operation multiple times in a single session. This causes the moment the operation occurs and the moment the device enters a stable visible state to be separated in the video timeline, and the separation time varies depending on the device type and operation method.

[0004] Due to the lack of leading parameters and time window modeling for the aforementioned waiting time, the system's attention and scoring are prone to being evenly distributed across the entire time axis or being attracted by instantaneous interference, causing peaks to fall at irrelevant changes before and after the operation. At the same time, the sequence of steps in the text description is based on an ideal process, which often differs from the actual sequence in real videos, further amplifying the problem of incorrect time period selection. Once evidence is taken from the wrong time period, the question-and-answer module will generate answers based on unreliable footage, ultimately resulting in repeated errors in judging causal relationships, temporal sequence, and whether the operation was completed. The core problem lies in the lack of a time leading parameter and corresponding time window modeling mechanism to constrain the evidence collection period. Ultimately, this leads to frequent ambiguities in question-and-answer sessions and poor conclusion stability in after-sales video question-and-answer scenarios. Summary of the Invention

[0005] This invention provides an intelligent after-sales customer service method, system, and storage medium, which solves the technical problems mentioned in the background.

[0006] This invention provides an intelligent after-sales customer service method, comprising the following steps:

[0007] Step S101: Perform feature extraction and temporal modeling on the video frame sequence to obtain the temporal hidden state sequence, and determine the operation time and operation type accordingly.

[0008] Step S102: Based on the device metadata and operation type, predict the delay distribution, take the lower and upper quantiles and combine them with the minimum inertia time and the maximum reasonable waiting time to determine the upper and lower boundaries of the time window based on the operation time.

[0009] Step S103: Calculate the time prior density based on the time delay distribution, construct a time prior mask by combining the upper and lower bounds of the time window, and simultaneously inject the time prior density and the time prior mask into the attention mechanism of the temporal neural network.

[0010] Step S104: On the injected temporal representation, search for the event saliency peak within the response inertial window, and determine the event time through sub-pixel refinement;

[0011] Step S105: Determine the time neighborhood weighted bandwidth based on the time delay distribution, aggregate time-series evidence within the response inertia window centered on the event time, and combine it with the work order information through the decoding module to output the answer probability distribution;

[0012] Step S106: Use the maximum probability of the answer probability distribution as the answer confidence level, and at the same time obtain the video observation duration. When the answer confidence level is lower than the lower limit of the answer probability or the observation duration is less than the upper limit of the time window, output the additional shooting duration; otherwise, output the customer service answer.

[0013] Further, determine the timing and type of operation, including the following steps:

[0014] Step S201: Extract features from each frame in the video frame sequence to obtain frame features, and input all frame features into a temporal neural network to obtain a temporal hidden state sequence.

[0015] Step S202: The product of the operation time discriminant vector and each time hidden state in the time-series hidden state sequence is calculated by the Sigmoid function to obtain the operation time confidence score, and the time with the largest operation time confidence score is selected as the operation time.

[0016] Step S203: Construct a fixed time neighborhood centered on the operation time. This neighborhood is the interval from the operation time minus half the width of the time neighborhood to the operation time plus half the width of the time neighborhood, and falls within the time range of the video frame sequence.

[0017] Step S204: Use the average pooling operator to perform an arithmetic average of the temporal hidden states in the neighborhood to obtain a temporal summary of the operation's vicinity.

[0018] Step S205: Encode the text question or work order content to obtain a text vector;

[0019] Step S206: The text vector is concatenated with the temporal summary near the operation. The operation type classification parameter matrix and the Softmax function are used to calculate and select the category with the highest probability as the operation type.

[0020] Furthermore, the device metadata is encoded to obtain the device embedding; the operation type is embedded to obtain the operation embedding.

[0021] The device embedding, operation embedding, and operation proximity timing convergence results are input into the learnable map. The shape and scale parameters are calculated using the Softplus function. The Softplus function is calculated by taking the exponent of the input value, adding 1, and then taking the natural logarithm. This calculation makes both the shape and scale parameters positive.

[0022] The waiting time distribution is established as a gamma distribution based on shape and scale parameters. The statistical lower bound increment and statistical upper bound increment corresponding to the lower and upper quantiles are calculated by using quantile functions. The statistical lower bound increment is compared with the minimum inertia time, and the maximum value of the two is added to the operation time to obtain the lower bound of the time window. The statistical upper bound increment is compared with the maximum reasonable waiting time, and the minimum value of the two is added to the operation time to obtain the upper bound of the time window.

[0023] Furthermore, based on shape and scale parameters, a gamma probability density function is determined. The difference between each time position and the operation time is calculated, and the temporal prior density of each time position is obtained through the gamma probability density function. For each time position, it is determined whether it falls within the interval from the lower to the upper bound of the time window. If it falls within this interval, the temporal prior mask of the corresponding time position is set to 1; otherwise, it is set to 0. A learnable non-negative scalar is set. The natural logarithm of the temporal prior density of each time position is multiplied by the learnable non-negative scalar. The natural logarithm of the temporal prior mask of each time position is also multiplied. The two results are added together to obtain the attention bias of each time position.

[0024] Furthermore, when a time position is not within the interval between the lower and upper bounds of the time window, the natural logarithm of the temporal prior mask corresponding to that time position is negative infinity. A dot product operation is performed on the query vector and key vector at each time position. The result is divided by the square root of the vector dimension, and the attention bias of the corresponding time position is added. The results for all time positions are then normalized to obtain the attention weights. Finally, the attention weights at each time position are weighted and summed with the value vector of the corresponding time position to obtain the injected temporal representation.

[0025] Furthermore, a dot product operation is performed on each temporal representation in the injected temporal representation sequence and the event discrimination vector. The event saliency at each time position is calculated using the Sigmoid function. The event saliency at each time position is multiplied by the temporal prior mask of the corresponding time position to obtain the in-window event saliency at each time position. Within the interval from the lower bound of the time window plus one to the upper bound of the time window minus one, the time position with the largest in-window event saliency is found. The time corresponding to the time position with the largest in-window event saliency is obtained. The difference between the in-window event saliency of the next time position and the in-window event saliency of the previous time position is calculated. This difference is multiplied by the frame time interval to obtain the numerator. The in-window event saliency of the next time position is calculated by subtracting twice the in-window event saliency of the current time position and adding the in-window event saliency of the previous time position to obtain the denominator. The numerator is divided by twice the denominator to obtain the time correction value. This time correction value is added to the time corresponding to the largest in-window event saliency to obtain the event time.

[0026] Further, the product of the square root of the shape parameter and the scale parameter is calculated to obtain the temporal neighborhood weighted bandwidth; the difference between each time position and the event time is squared, the result is divided by twice the square of the temporal neighborhood weighted bandwidth, the negative value of the result is then exponentially calculated, and the result is multiplied by the temporal prior mask of the corresponding time position to obtain the unnormalized weight of each time position; the unnormalized weights of all time positions are summed to obtain the normalization factor; the unnormalized weight of each time position is divided by the normalization factor to obtain the temporal weight of each time position; the temporal weight of each time position is multiplied by the injected temporal representation of the corresponding time position, and all product results are summed to obtain the evidence representation; the evidence representation, the text question or work order content vector, the device meta-information vector, and the operation type vector are concatenated, and the concatenated result is multiplied by the decoding parameter matrix and then calculated using the Softmax function to obtain the answer probability distribution.

[0027] Furthermore, the category with the highest probability is selected from the answer probability distribution as the customer service answer; the maximum probability value in the answer probability distribution is extracted as the answer confidence level; if the answer confidence level is less than the lower limit of the answer probability or the video observation duration is less than the upper limit of the time window, the difference between the upper limit of the time window and the video observation duration is calculated; when the difference is greater than 0, the difference is used as the additional shooting duration, and when the difference is less than or equal to zero, the additional shooting duration is determined to be 0; the quantitative suggestion that additional shooting is needed up to the upper limit of the time window and the additional shooting duration are output; if the answer confidence level is not less than the lower limit of the answer probability and the video observation duration is not less than the upper limit of the time window, the customer service answer is output.

[0028] This invention provides an intelligent after-sales customer service system, comprising:

[0029] The operation start point anchoring module is used to extract features and perform temporal modeling on the video frame sequence to obtain the temporal hidden state sequence, and determine the operation time and operation type accordingly.

[0030] The delay distribution prediction module is used to predict the delay distribution based on the device metadata and operation type. It takes the lower and upper quantiles and combines them with the minimum inertia time and the maximum reasonable waiting time to determine the upper and lower boundaries of the time window based on the operation time.

[0031] The prior mask injection module is used to calculate the temporal prior density based on the time delay distribution, construct the temporal prior mask by combining the upper and lower bounds of the time window, and inject the temporal prior density and the temporal prior mask into the attention mechanism of the temporal neural network at the same time.

[0032] The event localization module is used to search for the event saliency peak within the response inertial window on the injected time sequence representation, and determine the event time through sub-pixel refinement.

[0033] The answer probability distribution output module is used to determine the time neighborhood weighted bandwidth based on the time delay distribution, aggregate time-series evidence within the response inertia window centered on the event time, and output the answer probability distribution by combining it with the work order information through the decoding module.

[0034] The customer service answer output module uses the maximum probability of the answer probability distribution as the answer confidence level, and obtains the observation duration of the video. When the answer confidence level is lower than the lower limit of the answer probability or the observation duration is less than the upper limit of the time window, it outputs the additional shooting duration; otherwise, it outputs the customer service answer.

[0035] The present invention provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, it performs the above-described intelligent after-sales customer service method. The readable storage medium may be more specifically adopted and may include, but is not limited to, portable disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory, optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0036] The beneficial effects of this invention are as follows: By accurately modeling the response delay after device operation, this invention predicts the delay distribution and determines the time window based on device metadata and operation type. It injects time prior density and mask into the attention mechanism to constrain evidence retrieval to the effective time period, avoiding irrelevant interference and evidence collection during erroneous time periods. Combined with sub-pixel refinement, it achieves precise location of event moments and outputs reliable answers through temporal evidence aggregation and confidence assessment. At the same time, it provides quantitative suggestions for additional shooting when evidence is insufficient or the shooting does not cover the area. This process fundamentally solves the problem of frequent ambiguity and unstable conclusions caused by incorrect time period selection in after-sales video Q&A, greatly improving the accuracy and consistency of customer service answers, optimizing the user's after-sales consultation experience, and reducing the cost of ineffective communication. Attached Figure Description

[0037] Figure 1 This is a flowchart of an intelligent after-sales customer service method according to the present invention;

[0038] Figure 2 This is a flowchart illustrating the determination of the operation time and operation type according to the present invention;

[0039] Figure 3 This is a schematic diagram of an intelligent after-sales customer service system according to the present invention.

[0040] In the diagram: Operation starting point anchoring module 301, delay distribution prediction module 302, prior mask injection module 303, event localization module 304, answer probability distribution output module 305, customer service answer output module 306. Detailed Implementation

[0041] The subject matter described herein will now be discussed with reference to exemplary embodiments. It should be understood that these embodiments are discussed only to enable those skilled in the art to better understand and implement the subject matter described herein, and changes may be made to the function and arrangement of the elements discussed without departing from the scope of this specification. Various processes or components may be omitted, substituted, or added as needed in the examples. Furthermore, features described in some examples may be combined in other examples.

[0042] It should be noted that, unless otherwise defined, the technical or scientific terms used in one or more embodiments of the present invention should have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in one or more embodiments of the present invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" indicate that the element or object preceding the term encompasses the elements or objects listed following the term and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0043] like Figures 1-3 As shown, an intelligent after-sales customer service method includes the following steps:

[0044] Step S101: Perform feature extraction and temporal modeling on the video frame sequence to obtain the temporal hidden state sequence, and determine the operation time and operation type accordingly.

[0045] Step S102: Based on the device metadata and operation type, predict the delay distribution, take the lower and upper quantiles and combine them with the minimum inertia time and the maximum reasonable waiting time to determine the upper and lower boundaries of the time window based on the operation time.

[0046] Step S103: Calculate the time prior density based on the time delay distribution, construct a time prior mask by combining the upper and lower bounds of the time window, and simultaneously inject the time prior density and the time prior mask into the attention mechanism of the temporal neural network.

[0047] Step S104: On the injected temporal representation, search for the event saliency peak within the response inertial window, and determine the event time through sub-pixel refinement;

[0048] Step S105: Determine the time neighborhood weighted bandwidth based on the time delay distribution, aggregate time-series evidence within the response inertia window centered on the event time, and combine it with the work order information through the decoding module to output the answer probability distribution;

[0049] Step S106: Use the maximum probability of the answer probability distribution as the answer confidence level, and at the same time obtain the video observation duration. When the answer confidence level is lower than the lower limit of the answer probability or the observation duration is less than the upper limit of the time window, output the additional shooting duration; otherwise, output the customer service answer.

[0050] In one embodiment of the present invention, such as Figure 2 As shown, determining the operation time and operation type includes the following steps:

[0051] Step S201: Extract features from each frame in the video frame sequence to obtain frame features, and input all frame features into a temporal neural network to obtain a temporal hidden state sequence.

[0052] Step S202: The product of the operation time discriminant vector and each time hidden state in the time-series hidden state sequence is calculated by the Sigmoid function to obtain the operation time confidence score, and the time with the largest operation time confidence score is selected as the operation time.

[0053] Step S203: Construct a fixed time neighborhood centered on the operation time. This neighborhood is the interval from the operation time minus half the width of the time neighborhood to the operation time plus half the width of the time neighborhood, and falls within the time range of the video frame sequence.

[0054] Step S204: Use the average pooling operator to perform an arithmetic average of the temporal hidden states in the neighborhood to obtain a temporal summary of the operation's vicinity.

[0055] Step S205: Encode the text question or work order content to obtain a text vector;

[0056] Step S206: The text vector is concatenated with the temporal summary near the operation. The operation type classification parameter matrix and the Softmax function are used to calculate and select the category with the highest probability as the operation type.

[0057] It should be noted that pre-trained convolutional neural networks (CNNs) can be used for frame feature extraction, such as ResNet-50. First, the video frames are scaled to a fixed size (e.g., 224×224 pixels) and input into ResNet-50. The output of its last convolutional layer is then used as the frame features. For example, the frame features can be represented by a vector with a dimension of 2048. Other CNNs that can be used include MobileNet and EfficientNet. Long Short-Term Memory (LSTM) networks can also be used as temporal neural networks. The number of hidden units in an LSTM is the same as the number of frame features. Each hidden unit takes the frame features at the corresponding position as input and outputs the hidden state. For example, the hidden state is represented by a vector with a dimension of 512. All hidden states form a temporal hidden state sequence. In addition, the temporal neural network can also be GRU, Transformer, etc. The operation time discriminant vector is a learnable parameter vector with the same dimension as the temporal hidden state. The high-dimensional temporal hidden state can be compressed into a scalar by inner product calculation, and then mapped to the range of 0 to 1 by the Sigmoid function to obtain the confidence score that the time is the operation time. That is, the closer the score is to 1, the more likely it is to be the operation time.

[0058] It should be noted that user operations are usually instantaneous actions, such as pressing a button or unplugging a power cord. The visual changes are concentrated in a very short time. If the half-width is too small (e.g., 1 to 2 frames), keyframes before and after the operation may be missed; if it is too large (e.g., more than 10 frames), noise from irrelevant frames will be introduced. Therefore, setting the temporal neighborhood half-width to a range of 3 to 7 frames balances information integrity and resilience, and is suitable for most device operation scenarios, such as home appliances and electronic devices. Encoding textual questions or work order content involves converting the textual content into a fixed-dimensional vector. Common methods include encoding based on pre-trained language models (e.g., BERT, RoBERTa) or encoding based on word vectors (e.g.,...). The Word2Vec concatenation is used; the operation type classification parameter matrix is ​​also a learnable parameter, which maps the concatenated vector of the text vector and the operation's nearby temporal summary to the operation type space, such as categories like power off and long press reset. Its size is the concatenated vector dimension × the number of operation types. For example, if the text vector dimension is 768 and the operation's nearby temporal summary dimension is 512, the concatenated dimension is 1280. If there are 10 operation types, then the matrix dimension is 1280 × 10. In addition, the initial values ​​of the matrix can adopt a random normal distribution, i.e., mean 0 and standard deviation 0.01. During training, backpropagation is used for optimization to ensure that different operation types can be accurately distinguished, which will not be elaborated here.

[0059] In one embodiment of the present invention, device metadata is encoded to obtain device embedding; operation type is embedded to obtain operation embedding;

[0060] The device embedding, operation embedding, and operation proximity timing convergence results are input into the learnable map. The shape and scale parameters are calculated using the Softplus function. The Softplus function is calculated by taking the exponent of the input value, adding 1, and then taking the natural logarithm. This calculation makes both the shape and scale parameters positive.

[0061] The waiting time distribution is established as a gamma distribution based on shape and scale parameters. The statistical lower bound increment and statistical upper bound increment corresponding to the lower and upper quantiles are calculated by using quantile functions. The statistical lower bound increment is compared with the minimum inertia time, and the maximum value of the two is added to the operation time to obtain the lower bound of the time window. The statistical upper bound increment is compared with the maximum reasonable waiting time, and the minimum value of the two is added to the operation time to obtain the upper bound of the time window.

[0062] It should be noted that device metadata represents structured information describing the inherent attributes of a device, specifically including but not limited to: device model (e.g., smart door lock V3), hardware version (e.g., motherboard version 2.1), firmware version (e.g., firmware V4.5.2), core component type (e.g., motor model YY-300), and device category (e.g., home appliance / industrial controller / smart terminal). This information needs to be input into the encoding process in text form to distinguish the physical delay characteristics of different devices (e.g., the delay difference between mechanical locks and electronic switches). The learnable mapping is a small neural network used to embed device, operation, and operation proximity timing. The concatenated vector of the converged result is converted into parameters of a gamma distribution (shape parameter and scale parameter). For example, its structure is an input layer (dimension equal to the sum of the dimensions of the three vectors), 1 to 2 hidden layers (each hidden layer has a dimension of 1 / 2 of the input layer and uses the ReLU activation function), and an output layer (dimension 2, corresponding to the original output of the shape parameter and scale parameter respectively, which needs to be processed to positive values ​​by the Softplus function). For example, if the device embedding dimension is 128, the operation embedding dimension is 64, and the temporal converged result dimension is 256, then the input layer dimension is 128+64+256=448, the hidden layer can be set to 224 dimensions, and the output layer is 2 dimensions.

[0063] It should be noted that the quantile function is used to calculate the corresponding delay time based on the set quantiles (lower quantile and upper quantile). Preferably, the lower quantile is set to 0.05 to 0.1, and the upper quantile is set to 0.9 to 0.95. For example, the lower quantile is set to 0.1, and the upper quantile is set to 0.9. If the lower quantile is too small (e.g., 0.01), the statistical lower bound increment may be too small, causing the lower bound of the time window to be lower than the physically reasonable value. If it is too large (e.g., 0.2), it may miss the early reasonable delay. The same applies to the upper quantile, which is 0.9 to 0. The 95 range can cover 90% to 95% of historical latency data, ensuring that most normal latency is included while avoiding excessively wide windows due to extreme values; the minimum inertia time needs to match the physical characteristics of the device, for example, 0.1 to 0.3 seconds for electronic devices (such as mobile phones and routers) and 0.5 to 1 second for mechanical devices (such as door locks and valves); the maximum reasonable waiting time needs to consider the device's functional scenario, for example, 10 to 30 seconds for consumer-grade devices (such as home appliances) and 30 to 60 seconds for industrial-grade devices (such as controllers and motors).

[0064] In one embodiment of the present invention, a gamma probability density function is determined based on shape parameters and scale parameters. The difference between each time position and the operation time is calculated, and the temporal prior density of each time position is obtained through the gamma probability density function. For each time position, it is determined whether it falls within the interval from the lower bound to the upper bound of the time window. If it falls within the interval, the temporal prior mask of the corresponding time position is set to 1; otherwise, it is set to 0. A learnable non-negative scalar is set. The natural logarithm of the temporal prior density of each time position is multiplied by the learnable non-negative scalar. The natural logarithm of the temporal prior mask of each time position is also multiplied. The two results are added together to obtain the attention bias of each time position.

[0065] It should be noted that each time position represents a discrete time step corresponding to the video frame sequence, identified by an integer index. For example, the first frame corresponds to time position 1, the second frame corresponds to time position 2, and so on. The value range is all integers from 1 to the time sequence length. The correspondence between each time position and the actual time is determined by the video frame rate. That is, for a video of 30 frames per second, time position n corresponds to the actual time n / 30 seconds. When calculating the difference between the time position and the operation time, if the index of the time position is less than the index of the operation time, that is, the time position occurs before the operation, the difference is negative. Since the gamma distribution is only defined for non-negative values, the time prior density of the time position is directly set to 0. The initial value of the learnable non-negative scalar is set to 0.5 to 2, and adaptively adjusted to around 1 during training. The final value range is controlled between 0.1 and 5. The initial range of 0.5 to 2 can ensure that the prior plays an effective guiding role, and can also adapt to the differences in different devices / operations through training. When it finally converges to around 1, a balance between the prior and the data features can be achieved.

[0066] In one embodiment of the present invention, when a time position is not within the interval from the lower bound to the upper bound of the time window, the natural logarithm of the time prior mask corresponding to that time position is negative infinity; the query vector and the key vector of each time position are subjected to a dot product operation, the result is divided by the square root of the vector dimension, and the attention bias of the corresponding time position is added; the results of all time positions are normalized to obtain the attention weights; the attention weights of each time position are weighted and summed with the value vector of the corresponding time position to obtain the injected temporal representation.

[0067] It should be noted that the normalization operation specifically refers to Softmax normalization, which involves exponentializing the calculation results (dot product divided by the square root of the vector dimension plus attention bias) for all time positions, and then dividing by the sum of the exponentialized results for all time positions, so that the final sum of attention weights is 1, and the weight of each time position is between 0 and 1. The query vector, key vector, and value vector are all generated from the temporal representations obtained in the previous steps through three independent linear transformations. For the temporal representation of each time position, multiplying it by three learnable matrices (query matrix, key matrix, and value matrix) respectively yields the query vector, key vector, and value vector corresponding to that time position. Dividing the dot product result by the square root of the vector dimension is to avoid the dot product result being too large due to the high vector dimension. If the dot product result is too large, numerical saturation will occur after exponentialization, leading to gradient vanishing and affecting model training. Dividing by the square root controls the scale of the dot product result within a reasonable range (usually independent of the vector dimension), ensuring the stability of the Softmax operation. Preferably, the vector dimension is set to 256.

[0068] In one embodiment of the present invention, a dot product operation is performed on each time sequence representation in the injected time sequence and the event discrimination vector, and the event saliency at each time position is calculated using the Sigmoid function; the event saliency at each time position is multiplied by the temporal prior mask of the corresponding time position to obtain the in-window event saliency at each time position; within the interval from the lower bound of the time window plus one to the upper bound of the time window minus one, the time position with the largest in-window event saliency is found; the time of the time position corresponding to the largest in-window event saliency is obtained, and the difference between the in-window event saliency at the next time position and the in-window event saliency at the previous time position is calculated, and the difference is multiplied by the frame time interval to obtain the numerator; the in-window event saliency at the next time position is calculated by subtracting twice the in-window event saliency at the next time position and adding the in-window event saliency at the previous time position to obtain the denominator; the numerator is divided by twice the denominator to obtain the time correction value, and the time correction value is added to the time corresponding to the largest in-window event saliency to obtain the event time.

[0069] It should be noted that if the peak position falls within the lower or upper bound of the time window, its adjacent positions may exceed the window, resulting in a significance of 0, which would distort the interpolation results. Therefore, limiting the time to the interval between the lower bound plus one and the lower bound minus one ensures that there is data support before and after the peak position, improving the accuracy of time refinement. When calculating the denominator, there may be extreme cases where the result is zero, such as when the significance before and after the peak position is completely symmetrical and equal to the peak. In this case, the time corresponding to the peak position is directly used as the event time without time correction. This approach avoids division errors, and when the denominator is zero, the parabola is approximately a horizontal straight line, and the peak position itself can accurately represent the event. At the event moment, correction is meaningless; the dimension of the event discrimination vector must be completely consistent with the dimension of a single time sequence representation in the time sequence. For example, if each injected time sequence representation is a 256-dimensional vector, then the event discrimination vector must also be set to 256 dimensions; the initial value of the event discrimination vector adopts a random normal distribution, that is, the mean is 0 and the standard deviation is 0.01. During training, it is adaptively optimized through backpropagation. The core function of the event discrimination vector is to capture the features related to stable visible events in the time sequence representation, such as the visual features of an LED light turning on or a door lock unlocking; the frame interval is determined according to the video frame rate. For example, for a 30 fps video, 1 / 30 second (approximately 0.033 seconds) is used.

[0070] In one embodiment of the present invention, the product of the square root of the shape parameter and the scale parameter is calculated to obtain the temporal neighborhood weighted bandwidth; the difference between each time position and the event time is squared, the result is divided by twice the square of the temporal neighborhood weighted bandwidth, the negative value of the result is taken and then exponentially calculated with the natural constant as the base, and the result is multiplied by the temporal prior mask of the corresponding time position to obtain the unnormalized weight of each time position; the unnormalized weights of all time positions are summed to obtain the normalization factor; the unnormalized weight of each time position is divided by the normalization factor to obtain the temporal weight of each time position; the temporal weight of each time position is multiplied by the injected temporal representation of the corresponding time position, and all product results are summed to obtain the evidence representation; the evidence representation, the text question or work order content vector, the device meta-information vector, and the operation type vector are concatenated, and the concatenation result is multiplied by the decoding parameter matrix and then calculated using the Softmax function to obtain the answer probability distribution.

[0071] It should be noted that vector concatenation must follow a fixed order. The recommended order is: evidence representation, text question or work order content vector, device metadata vector, and operation type vector. The fixed order is to ensure that the dimensions of the concatenated vectors are arranged in a consistent manner, so that the training of the decoding parameter matrix has a clear mapping relationship. The initial values ​​of the decoding parameter matrix also adopt a random normal distribution, and the matrix dimension is the total dimension of the concatenated vectors × the number of answer categories. Adaptive optimization is performed through backpropagation during training, which will not be elaborated here.

[0072] In one embodiment of the present invention, the category with the highest probability is selected as the customer service answer from the answer probability distribution; the maximum probability value in the answer probability distribution is extracted as the answer confidence level; if the answer confidence level is less than the lower limit of the answer probability or the video observation duration is less than the upper limit of the time window, the difference between the upper limit of the time window and the video observation duration is calculated; when the difference is greater than 0, the difference is used as the additional shooting duration, and when the difference is less than or equal to zero, the additional shooting duration is determined to be 0; a quantitative suggestion that additional shooting is needed to the upper limit of the time window and the additional shooting duration are output; if the answer confidence level is not less than the lower limit of the answer probability and the video observation duration is not less than the upper limit of the time window, the customer service answer is output.

[0073] It should be noted that, in addition to outputting customer service answers, the system can also simultaneously output the event time and the upper and lower bounds of the time window, clearly defining the event occurrence time corresponding to the final answer and the time window on which the judgment is based, ensuring the traceability and explainability of the answer; the video observation duration represents the total actual time of the video from the start to the end of the recording; supplementary recording must start from the end time of the current video and continue recording until the time point corresponding to the upper bound of the time window, ensuring that the supplementary recording segment is seamlessly connected with the original video and fully covers the interval from the current end time to the upper bound of the time window; for example, if the original video observation duration is 10 seconds and the upper bound of the time window is 15 seconds, then 5 seconds of supplementary recording will be used to make the total recording duration reach 15 seconds, fully covering the response inertia window; the lower limit of the answer probability is a preset parameter, preferably set to 0.75.

[0074] It should be noted that, taking a user-uploaded after-sales video and work order content for a certain model of smart door lock as an example, the work order content inquires whether the door lock successfully resets and unlocks after pressing and holding the reset button. The device metadata includes the smart door lock model and firmware version. After processing by the method of this invention, step S101 determines the operation time as the moment when the user presses and holds the reset button in the video, and the operation type is long press reset; step S102 predicts the delay distribution and determines the time window as 2 to 8 seconds after the operation time; steps S103 to S104 locate the event time as 5 seconds after the operation time, corresponding to the stable event of the door lock unlock indicator light being constantly on; step S105 outputs the probability distribution of the answer, in which the probability of the smart door lock successfully resetting and unlocking is the highest; step S106 determines that the confidence level of the answer is higher than the lower limit and the video observation duration covers the upper limit of the time window, and the final output customer service answer is that the smart door lock of this model has successfully completed the long press reset operation, and the door lock unlock indicator light is constantly on 5 seconds after the operation, and the door lock is currently in the unlocked state.

[0075] In one embodiment of the present invention, such as Figure 3 As shown, an intelligent after-sales customer service system includes:

[0076] The operation start point anchoring module 301 is used to extract features and perform temporal modeling on the video frame sequence to obtain the temporal hidden state sequence, and determine the operation time and operation type accordingly.

[0077] The delay distribution prediction module 302 is used to predict the delay distribution based on the device metadata and operation type, take the lower and upper quantiles and combine them with the minimum inertia time and the maximum reasonable waiting time to determine the upper and lower boundaries of the time window based on the operation time.

[0078] The prior mask injection module 303 is used to calculate the time prior density based on the time delay distribution, construct the time prior mask by combining the upper and lower bounds of the time window, and inject the time prior density and the time prior mask into the attention mechanism of the temporal neural network at the same time.

[0079] The event localization module 304 is used to search for the event saliency peak within the response inertial window on the injected time sequence representation, and determine the event time through sub-pixel refinement.

[0080] The answer probability distribution output module 305 is used to determine the time neighborhood weighted bandwidth based on the time delay distribution, aggregate time-series evidence within the response inertia window centered on the event time, and output the answer probability distribution by combining it with the work order information through the decoding module.

[0081] The customer service answer output module 306 is used to take the maximum probability of the answer probability distribution as the answer confidence level, and at the same time obtain the observation duration of the video. When the answer confidence level is lower than the lower limit of the answer probability or the observation duration is less than the upper limit of the time window, the additional shooting duration is output; otherwise, the customer service answer is output.

[0082] In one embodiment of the present invention, the present invention provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, it performs the above-described intelligent after-sales customer service method. The readable storage medium may be more specifically adopted and may include, but is not limited to, portable disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory, optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0083] It should be noted that the interval and threshold sizes are set for ease of comparison. The size of the threshold depends on the amount of sample data and the base number set by those skilled in the art for each set of sample data, as long as it does not affect the proportional relationship between the parameter and the quantized value. Furthermore, the above formulas are all dimensionless calculations, and the formulas are derived from software simulations using a large amount of collected data to obtain the most recent real-world results. The preset parameters in the formulas are set by those skilled in the art according to the actual situation.

[0084] The embodiments of this example have been described above. However, this example is not limited to the specific implementation methods described above. The specific implementation methods described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms based on the guidance of this example, and all of them are within the protection scope of this example.

Claims

1. An intelligent after-sales customer service method, characterized in that, Includes the following steps: Step S101: Perform feature extraction and temporal modeling on the video frame sequence to obtain the temporal hidden state sequence, and determine the operation time and operation type accordingly. Step S102: Based on the device metadata and operation type, predict the delay distribution, take the lower and upper quantiles and combine them with the minimum inertia time and the maximum reasonable waiting time to determine the upper and lower boundaries of the time window based on the operation time. Step S103: Calculate the time prior density based on the time delay distribution, construct a time prior mask by combining the upper and lower bounds of the time window, and simultaneously inject the time prior density and the time prior mask into the attention mechanism of the temporal neural network. Step S104: On the injected temporal representation, search for the event saliency peak within the response inertial window, and determine the event time through sub-pixel refinement; Step S105: Determine the time neighborhood weighted bandwidth based on the time delay distribution, aggregate time-series evidence within the response inertia window centered on the event time, and combine it with the work order information through the decoding module to output the answer probability distribution; Step S106: Use the maximum probability of the answer probability distribution as the answer confidence level, and at the same time obtain the video observation duration. When the answer confidence level is lower than the lower limit of the answer probability or the observation duration is less than the upper limit of the time window, output the additional shooting duration; otherwise, output the customer service answer.

2. The intelligent after-sales customer service method according to claim 1, characterized in that, Determine the timing and type of operation, including the following steps: Step S201: Extract features from each frame in the video frame sequence to obtain frame features, and input all frame features into a temporal neural network to obtain a temporal hidden state sequence. Step S202: The product of the operation time discriminant vector and each time hidden state in the time-series hidden state sequence is calculated by the Sigmoid function to obtain the operation time confidence score, and the time with the largest operation time confidence score is selected as the operation time. Step S203: Construct a fixed time neighborhood centered on the operation time. This neighborhood is the interval from the operation time minus half the width of the time neighborhood to the operation time plus half the width of the time neighborhood, and falls within the time range of the video frame sequence. Step S204: Use the average pooling operator to perform an arithmetic average of the temporal hidden states in the neighborhood to obtain a temporal summary of the operation's vicinity. Step S205: Encode the text question or work order content to obtain a text vector; Step S206: The text vector is concatenated with the temporal summary near the operation. The operation type classification parameter matrix and the Softmax function are used to calculate and select the category with the highest probability as the operation type.

3. The intelligent after-sales customer service method according to claim 1, characterized in that, Encode the device metadata to obtain the device embedding; embed the operation type to obtain the operation embedding; The device embedding, operation embedding, and operation proximity timing convergence results are input into the learnable map. The shape and scale parameters are calculated using the Softplus function. The Softplus function is calculated by taking the exponent of the input value, adding 1, and then taking the natural logarithm. This calculation makes both the shape and scale parameters positive. The waiting time distribution is established as a gamma distribution based on shape and scale parameters. The statistical lower bound increment and statistical upper bound increment corresponding to the lower and upper quantiles are calculated by using quantile functions. The statistical lower bound increment is compared with the minimum inertia time, and the maximum value of the two is added to the operation time to obtain the lower bound of the time window. The statistical upper bound increment is compared with the maximum reasonable waiting time, and the minimum value of the two is added to the operation time to obtain the upper bound of the time window.

4. The intelligent after-sales customer service method according to claim 1, characterized in that, The gamma distribution probability density function is determined based on shape and scale parameters. The difference between each time position and the operation time is calculated, and the time prior density of each time position is obtained through the gamma distribution probability density function. For each time position, determine whether it falls within the interval from the lower bound to the upper bound of the time window. If it falls within this interval, the temporal prior mask for the corresponding time position is set to 1; otherwise, it is set to 0. Set a learnable non-negative scalar, take the natural logarithm of the temporal prior density for each time position, multiply it by the learnable non-negative scalar, take the natural logarithm of the temporal prior mask for each time position, and add the two results to obtain the attention bias for each time position.

5. The intelligent after-sales customer service method according to claim 4, characterized in that, When the time position is not within the interval from the lower bound to the upper bound of the time window, the natural logarithm of the time prior mask corresponding to that time position is negative infinity; perform a dot product operation on the query vector and the key vector of each time position, divide the result by the square root of the vector dimension, add the attention bias of the corresponding time position, and normalize the results of all time positions to obtain the attention weight. The attention weight at each time point is summed with the value vector at the corresponding time point to obtain the injected temporal representation.

6. The intelligent after-sales customer service method according to claim 1, characterized in that, Each time-series representation in the injected temporal representation sequence is multiplied by the event discrimination vector, and the event saliency at each time position is calculated using the Sigmoid function. The event saliency at each time position is multiplied by the temporal prior mask of the corresponding time position to obtain the in-window event saliency at each time position. Within the interval from the lower bound of the time window plus one to the upper bound of the time window minus one, the time position with the largest in-window event saliency is found. The time of the time position corresponding to the largest in-window event saliency is obtained, and the difference between the in-window event saliency of the next time position and the in-window event saliency of the previous time position is calculated. This difference is multiplied by the frame time interval to obtain the numerator. The in-window event saliency of the next time position is calculated by subtracting twice the in-window event saliency of the current time position and adding the in-window event saliency of the previous time position to obtain the denominator. The numerator is divided by twice the denominator to obtain the time correction value, and this time correction value is added to the time corresponding to the largest in-window event saliency to obtain the event time.

7. The intelligent after-sales customer service method according to claim 1, characterized in that, The product of the square root of the shape parameter and the scale parameter is calculated to obtain the temporal neighborhood weighted bandwidth. The difference between each time position and the event time is squared, the result is divided by twice the square of the temporal neighborhood weighted bandwidth, the negative value of the result is taken and then exponentially calculated. The result is multiplied by the temporal prior mask of the corresponding time position to obtain the unnormalized weight of each time position. The unnormalized weights of all time positions are summed to obtain the normalization factor. Divide the unnormalized weight of each time position by the normalization factor to obtain the time weight of each time position; The time weight of each time position is multiplied by the injected time series representation of the corresponding time position, and all product results are summed to obtain the evidence representation. The evidence representation, text question or work order content vector, device meta-information vector and operation type vector are concatenated, and the concatenation result is multiplied by the decoding parameter matrix and then calculated by the Softmax function to obtain the answer probability distribution.

8. The intelligent after-sales customer service method according to claim 1, characterized in that, Select the category with the highest probability from the answer probability distribution as the customer service answer; extract the maximum probability value from the answer probability distribution as the answer confidence level; if the answer confidence level is less than the lower limit of the answer probability or the video observation duration is less than the upper limit of the time window, calculate the difference between the upper limit of the time window and the video observation duration; When the difference is greater than 0, the difference is used as the additional shooting time; when the difference is less than or equal to zero, the additional shooting time is set to 0. The output needs to include quantitative suggestions for shooting up to the upper limit of the time window and additional shooting time; If the confidence level of the answer is not less than the lower probability limit of the answer and the video observation duration is not less than the upper bound of the time window, output the customer service answer.

9. An intelligent after-sales customer service system, characterized in that, Performing an intelligent after-sales customer service method as described in any one of claims 1 to 8 includes: The operation start point anchoring module is used to extract features and perform temporal modeling on the video frame sequence to obtain the temporal hidden state sequence, and determine the operation time and operation type accordingly. The delay distribution prediction module is used to predict the delay distribution based on the device metadata and operation type. It takes the lower and upper quantiles and combines them with the minimum inertia time and the maximum reasonable waiting time to determine the upper and lower boundaries of the time window based on the operation time. The prior mask injection module is used to calculate the temporal prior density based on the time delay distribution, construct the temporal prior mask by combining the upper and lower bounds of the time window, and inject the temporal prior density and the temporal prior mask into the attention mechanism of the temporal neural network at the same time. The event localization module is used to search for the event saliency peak within the response inertial window on the injected time sequence representation, and determine the event time through sub-pixel refinement. The answer probability distribution output module is used to determine the time neighborhood weighted bandwidth based on the time delay distribution, aggregate time-series evidence within the response inertia window centered on the event time, and output the answer probability distribution by combining it with the work order information through the decoding module. The customer service answer output module uses the maximum probability of the answer probability distribution as the answer confidence level, and obtains the observation duration of the video. When the answer confidence level is lower than the lower limit of the answer probability or the observation duration is less than the upper limit of the time window, it outputs the additional shooting duration; otherwise, it outputs the customer service answer.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it performs an intelligent after-sales customer service method as described in any one of claims 1 to 8.

Images