Electric power marketing service behavior supervision and control method and system

Abstract

The invention discloses an electric power marketing service behavior supervision and control method and system, and relates to the technical field of behavior supervision, and the method comprises the steps: collecting audio, video and position information in an electric power access service process, carrying out the multi-modal data fusion analysis of the collected information, and comprehensively judging whether the service behavior is compliant or not; the wearing condition of the customer manager is monitored in real time through the AI image recognition technology, and whether the behavior is correct or not is judged based on the fused data; performing encrypted storage and non-tampering recording on behavior data of a customer manager in each service link based on a block chain technology, and performing intelligent labeling and tracking on key operation steps in a service process; and verifying the audio and video data provided by the customer manager through a deep counterfeiting detection technology, and outputting a final verification result. According to the invention, by introducing a multi-modal data fusion technology, an intelligent behavior analysis technology and a block chain storage technology, brand new improvement of the power management and control system in customer manager service behavior supervision is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of behavior supervision, and in particular to a method and system for supervising and controlling power marketing service behavior. Background Art

[0002] With the expansion of the power system and the growing demand for electricity, the behavior of account managers in the process of power access services is crucial to ensuring the compliance, efficiency and safety of power services. Account managers are involved in a variety of operations when performing services, including on-site surveys, design reviews, intermediate inspections and final acceptance. However, due to the complexity of manual operations, account managers may violate regulations during the service process, such as failing to wear safety equipment as required, improper communication, failing to complete on-site tasks according to standards, or not working at designated locations. These violations not only affect the quality of power services, but may also pose safety risks.

[0003] Existing power management and control systems rely on manual supervision and records from a single data source, such as text reports or basic video surveillance. This approach has many shortcomings: it is difficult for supervisors to monitor the actual behavior of account managers in real time and comprehensively; a single data source is susceptible to falsification or misleading; and supervision efficiency is low and violations are easily missed. As the scale and complexity of power services increase, there is an urgent need for an intelligent, multi-dimensional supervision method to conduct a comprehensive and accurate assessment of the behavior of account managers. Summary of the invention

[0004] In view of the above existing problems, the present invention is proposed.

[0005] Therefore, the present invention provides a method for supervising and controlling the behavior of power marketing services. In order to solve the problem that traditional power management and control methods lack real-time and all-round supervision of account managers, existing monitoring methods usually rely on a single data source, such as video or text reports, and it is difficult to obtain the real behavior of account managers when performing tasks on site from multiple angles. Secondly, the present invention can also solve the problem that the singleness of the data source of the prior art makes it difficult to combine multi-dimensional data such as audio, video, and location information for comprehensive analysis, which can easily lead to misjudgment or missed judgment. Furthermore, the existing system lacks intelligent behavior analysis means, and it is difficult to perform compliance detection of the voice, behavior, and operation position of the account manager in an automated manner. In addition, the present invention can solve the problem that the prior art has insufficient capabilities in forgery and tampering detection, and the account manager may bypass the supervision system by forging audio and video data or tampering with records, and lacks effective means to detect and prevent such behavior.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions, a method for supervising and controlling power marketing service behavior, comprising: Collect the audio, video and location information of the account manager during the power access service process, and perform multimodal data fusion analysis on the collected audio, video and location information to comprehensively judge whether the account manager's service behavior is compliant; use AI image recognition technology to monitor the account manager's wear in real time, and judge whether the behavior is correct based on the fused data; use blockchain technology to encrypt and store the account manager's behavior data in each service link and record it in an unalterable manner, and intelligently mark and track the key operation steps in the service process to accurately manage the account manager's operation process; verify the authenticity of the audio and video data provided by the account manager through deep counterfeit detection technology, and output the final verification result.

[0007] As a preferred solution of the method for supervising and controlling power marketing service behavior described in the present invention, the multimodal data fusion analysis is to achieve a comprehensive evaluation of the customer manager's behavior through a comprehensive analysis of audio, video and location information, including: performing speech recognition on audio data to convert speech into text; using natural language processing technology to perform semantic analysis on the converted text to identify whether the customer manager has any service violations such as using sensitive words and inappropriate language; performing image recognition on video data, combining location information to analyze whether the customer manager has correctly performed on-site operations, and verifying whether he or she provides services at the specified location.

[0008] As a preferred solution of the power marketing service behavior supervision and control method of the present invention, the speech recognition of audio data includes performing short-time Fourier transform on the collected audio signal and converting it into text representation using MFCC feature extraction: Get the Mel frequency cepstral coefficient features through the Mel filter: in, Represents the frequency spectrum of the audio signal at frequency f and time t; represents the time variable; f represents the frequency; t represents the time; represents the MFCC features at time t; M represents the number of Mel filters; Indicates the number of frequency band divisions; represents the frequency component; Represents the response value of the mth Mel filter; represents the integral representation of the audio signal; Represents the power spectrum of a frequency domain signal; represents the complex exponential function, and j represents the imaginary unit.

[0009] Convert audio features to text, i.e.: Among them, T is the text representation converted from the audio signal, and SR() represents the speech recognition function.

[0010] Through predefined vocabulary sets or SVM based on sentiment classification, we can identify whether there are sensitive words or inappropriate language. The detection results are passed through the classifier function Output: in, Indicates the result of audio violation detection, 0 indicates no violation, and 1 indicates a violation.

[0011] As a preferred solution of the method for supervising and controlling the behavior of power marketing services described in the present invention, the image recognition of video data includes inputting a video frame sequence and extracting behavioral features of the video frames using an architecture combining a convolutional neural network and a long short-term memory network.

[0012] For video signals, the optical flow features of lip motion are extracted by optical flow algorithm, which is expressed as: in, Is the video in position and the pixel value at time t, indicating the speed of lip movement; Represents the optical flow features extracted from the video data at time t.

[0013] The convolutional neural network is used to identify the behavior of the video motion features. The convolutional neural network extracts the spatial and temporal features in the video frame to detect whether the customer manager's behavior is compliant: in, Indicates the result of video behavior recognition, 0 indicates compliant behavior, and 1 indicates illegal behavior; Represents the convolutional neural network model used to classify video features.

[0014] As a preferred solution of the method for supervising and controlling power marketing service behavior described in the present invention, the comprehensive judgment of whether the customer manager's service behavior is compliant includes the combined use of semantic analysis and image recognition, through the synchronous analysis of audio, video and location information, and when inconsistent behavior is detected, immediately generating corresponding feedback information to remind the customer manager to adjust the behavior.

[0015] Through GPS positioning technology, the location information of the account manager is collected and expressed as the coordinate points of the time series: By calculating the preset location With actual location The Euclidean distance of: in, Indicates the preset location With actual location The Euclidean distance of , , They respectively represent the actual location information of the account manager at time t in spatial coordinates; , , They respectively represent the preset position information of the account manager in the spatial coordinates at time t.

[0016] like Less than a predetermined threshold , the account manager is considered to be at the specified location; the location matching flag is specifically represented by: The violation detection results of audio processing , video behavior recognition results And location information matching results Conduct multimodal fusion to comprehensively assess whether the account manager's behavior is compliant.

[0017] Multimodal fusion obtains the final judgment result by weighted summing the classification results of audio, video and location information. It is expressed as: in, They represent the weight coefficients of audio violation detection results, video behavior recognition results, and location information matching results respectively. The weight setting is optimized through training data.

[0018] like Exceeding the preset threshold , then the account manager’s behavior is judged to be non-compliant.

[0019] As a preferred solution of the method for supervising and controlling power marketing service behavior described in the present invention, the key operation steps include on-site investigation, design review, intermediate testing and final acceptance.

[0020] The intelligent labeling and tracking technology is used to automatically identify the service operations of customer managers during on-site surveys, design reviews, intermediate inspections, and final acceptance inspections, and to mark completed operation steps through the AI ​​system, verify whether key steps have not been omitted or tampered with, and record the completion of operations through the blockchain.

[0021] As a preferred solution of the power marketing service behavior supervision and control method described in the present invention, the deep fake detection technology includes: the synchronization detection of audio and video is detected by comparing the Mel frequency cepstral coefficients of the audio and the lip movement optical flow features in the video, and the synchronization measurement is specifically as follows: Audio Features and video features Matching is performed through dynamic time warping, and the matching degree is: in, is the Euclidean distance between feature vectors; N is the number of time points in the audio or video sequence; is the audio and video synchronization matching score; is the audio feature at the i-th time point; is the video optical flow feature at the i-th time point; if Less than threshold , the audio and video are considered out of sync and are judged to be forged.

[0022] By calculating the correlation between facial expressions and tone of voice emotional features, the matching degree is defined as: in, is the facial expression feature of the video at time t, is the audio tone and emotional feature at time t, and are the means of facial expression and tone of voice emotion features, Indicates the matching degree between facial expression and tone of voice; if Less than threshold , then there is forgery.

[0023] By calculating the abnormal points of the spectrum, the audio forgery traces are detected and the spectrum change amount is defined for: Detect abnormal changes in the spectrum and define the anomaly score as: in, is the audio spectrum anomaly score; is the average value of the spectrum variation at frequency f; represents the frequency spectrum of the audio signal at frequency f before time t; if Exceeding the threshold , the audio has been fabricated or edited.

[0024] The brightness distribution of a video frame is defined as the sum of the pixel brightness. The brightness changes are analyzed using Laplace transform by detecting the spatial and temporal consistency of the brightness changes: in, Indicates the brightness change of the i-th frame; represents the brightness distribution of video frame j at time t; C represents the number of frames; Indicates the consistency score of light and shadow changes; represents the second-order derivative of the brightness of the j-th frame of the video in space; if the difference in light and shadow changes Greater than threshold , then there is forgery.

[0025] When deep fake detection technology identifies forged or tampered audio or video data, the system automatically triggers an alarm mechanism, records the information to the blockchain and notifies senior management, while generating a complete event report for auditing and tracing.

[0026] As a preferred solution of the power marketing service behavior supervision and control system described in the present invention, the system includes: using multimodal data fusion technology to collect audio, video and location information of the customer manager in the power access service process in real time; the system obtains multi-source data of the customer manager by the data acquisition module, including voice conversations, on-site images and GPS location information; the audio processing module converts the audio signal into text and performs semantic analysis to detect sensitive words and inappropriate language; the video processing module detects the work flow and dress specifications of the customer manager through behavior recognition technology; the location matching module verifies whether the customer manager's work location complies with regulations through GPS positioning.

[0027] The system performs deep forgery detection through the analysis module. The audio and video synchronization detection module detects the synchronization of audio and video features by matching the time series of the two. The facial expression matching detection module compares the facial expressions in the video with the tone and emotional features in the audio to detect whether they are consistent. The video light and shadow consistency detection module determines whether the video has been edited by analyzing the brightness changes of the video frames. The audio waveform analysis module detects traces of forgery in the audio by extracting spectral features. The detection results of each module are comprehensively calculated through the multimodal data fusion module to generate a forgery detection score.

[0028] The blockchain storage and verification module encrypts all collected and analyzed data and stores them in the blockchain, and the intelligent labeling and tracking module labels and tracks the key operation steps in the service process.

[0029] A computer device includes a memory and a processor, wherein the memory stores a computer program, and wherein the processor implements the steps of a method for supervising and controlling power marketing service behaviors when executing the computer program.

[0030] A computer-readable storage medium having a computer program stored thereon, characterized in that when the computer program is executed by a processor, the steps of a method for supervising and controlling power marketing service behavior are implemented.

[0031] Beneficial effects of the present invention: Overall, the present invention can monitor the behavior of account managers in real time and in all aspects. By collecting audio, video and location information, the system can comprehensively and real-time obtain the behavior data of account managers in power access services, eliminating the monitoring blind spots of traditional systems. At the same time, multimodal data fusion analysis effectively solves the problem of single data source. The system can comprehensively analyze the audio processing results, video behavior recognition results and location information matching results, and accurately evaluate whether the behavior of account managers is compliant through weighted summation, which significantly improves the analysis accuracy and reduces misjudgments and missed judgments.

[0032] At the same time, the collaborative work of the various modules of the present invention further enhances the overall performance of the system. For example, the audio and video synchronization detection, facial expression matching, and light and shadow consistency detection modules cooperate with each other when processing audio and video data, ensuring that the system can not only analyze the behavior of the account manager, but also detect the authenticity of the audio and video data, thereby effectively preventing forgery and tampering. At the same time, by combining natural language processing technology and location matching technology, the system can intelligently analyze the voice and location information of the account manager, thereby conducting a more comprehensive analysis of the compliance of his behavior, ensuring the compliance of the entire service process.

[0033] In addition, the present invention introduces blockchain storage technology to encrypt all data and store it in the blockchain, ensuring the data's immutability and traceability. Even if disputes or security risks arise, the account manager's behavior can be audited by tracing the data on the chain, thereby improving the transparency of the supervision process and the reliability of the data. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0035] Figure 1 A flowchart of a method for supervising and controlling power marketing service behavior is provided for one embodiment of the present invention.

[0036] Figure 2 A schematic diagram of a power marketing service behavior supervision and control system provided for an embodiment of the present invention. DETAILED DESCRIPTION

[0037] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0038] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0039] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0040] The present invention is described in detail with reference to schematic diagrams. When describing the embodiments of the present invention, for the sake of convenience, the cross-sectional diagrams showing the device structure will not be partially enlarged according to the general scale, and the schematic diagrams are only examples, which should not limit the scope of protection of the present invention. In addition, in actual production, the three-dimensional dimensions of length, width and depth should be included.

[0041] At the same time, in the description of the present invention, it should be noted that the directions or positional relationships indicated by the terms "upper, lower, inner and outer" are based on the directions or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore cannot be understood as limiting the present invention. In addition, the terms "first, second or third" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0042] In the present invention, unless otherwise clearly specified and limited, the terms "install, connect, connect" should be understood in a broad sense, for example: it can be a fixed connection, a detachable connection or an integral connection; it can also be a mechanical connection, an electrical connection or a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0043] Example 1, reference Figure 1 , which is the first embodiment of the present invention, and provides a method for supervising and controlling power marketing service behavior, comprising: S1: Collect the audio, video and location information of the account manager during the power access service process, and perform multimodal data fusion analysis on the collected audio, video and location information to comprehensively determine whether the account manager's service behavior is compliant.

[0044] S2: Use AI image recognition technology to monitor the account manager's wear in real time, and judge whether the behavior is correct based on the fused data.

[0045] S3: Based on blockchain technology, the account manager’s behavioral data in each service link is encrypted and stored and recorded in an unalterable manner. At the same time, the key operation steps in the service process are intelligently marked and tracked to accurately manage the account manager’s operation process.

[0046] S4: Verify the authenticity of the audio and video data provided by the account manager through deep fake detection technology, and output the final verification result.

[0047] The multimodal data fusion analysis is to achieve a comprehensive assessment of the customer manager's behavior through a comprehensive analysis of audio, video and location information, including: Perform speech recognition on audio data and convert speech into text.

[0048] Use natural language processing technology to perform semantic analysis on the converted text to identify whether the account manager has violated service regulations by using sensitive words or inappropriate language.

[0049] Perform image recognition on video data and combine it with location information to analyze whether the account manager performs on-site operations correctly and verify whether he or she provides services at the specified location.

[0050] The speech recognition of the audio data includes performing short-time Fourier transform on the collected audio signal and converting it into text representation using MFCC feature extraction: Get the Mel frequency cepstral coefficient features through the Mel filter: in, Represents the frequency spectrum of the audio signal at frequency f and time t; represents the time variable; f represents the frequency; t represents the time; represents the MFCC features at time t; M represents the number of Mel filters; Indicates the number of frequency band divisions; represents the frequency component; Represents the response value of the mth Mel filter; represents the integral representation of the audio signal; Represents the power spectrum of a frequency domain signal; represents the complex exponential function, and j represents the imaginary unit.

[0051] Convert audio features to text, i.e.: Among them, T is the text representation converted from the audio signal, and SR() represents the speech recognition function.

[0052] Through predefined vocabulary sets or SVM based on sentiment classification, we can identify whether there are sensitive words or inappropriate language. The detection results are passed through the classifier function Output: in, Indicates the result of audio violation detection, 0 indicates no violation, and 1 indicates a violation.

[0053] The image recognition of video data includes inputting a video frame sequence and extracting behavioral features from the video frames using an architecture combining a convolutional neural network and a long short-term memory network.

[0054] For video signals, the optical flow features of lip motion are extracted by optical flow algorithm, which is expressed as: in, Is the video in position and the pixel value at time t, indicating the speed of lip movement; Represents the optical flow features extracted from the video data at time t.

[0055] The convolutional neural network is used to identify the behavior of the video motion features. The convolutional neural network extracts the spatial and temporal features in the video frame to detect whether the customer manager's behavior is compliant: in, Indicates the result of video behavior recognition, 0 indicates compliant behavior, and 1 indicates illegal behavior; Represents the convolutional neural network model used to classify video features.

[0056] The comprehensive judgment of whether the customer manager's service behavior is compliant includes the combined use of semantic analysis and image recognition, through the synchronous analysis of audio, video and location information, and when non-compliant behavior is detected, immediately generating corresponding feedback information to remind the customer manager to adjust the behavior.

[0057] Through GPS positioning technology, the location information of the account manager is collected and expressed as the coordinate points of the time series: By calculating the preset location With actual location The Euclidean distance of: in, Indicates the preset location With actual location The Euclidean distance of , , They respectively represent the actual location information of the account manager at time t in spatial coordinates; , , They respectively represent the preset position information of the account manager in the spatial coordinates at time t.

[0058] like Less than a predetermined threshold , the account manager is considered to be at the specified location; the location matching flag is specifically represented by: The violation detection results of audio processing , video behavior recognition results And location information matching results Conduct multimodal fusion to comprehensively assess whether the account manager's behavior is compliant.

[0059] Multimodal fusion obtains the final judgment result by weighted summing the classification results of audio, video and location information. It is expressed as: in, They represent the weight coefficients of audio violation detection results, video behavior recognition results, and location information matching results respectively. The weight setting is optimized through training data.

[0060] like Exceeding the preset threshold , then the account manager’s behavior is judged to be non-compliant.

[0061] The key operational steps include on-site investigation, design review, intermediate testing and final acceptance.

[0062] The intelligent labeling and tracking technology is used to automatically identify the service operations of customer managers during on-site surveys, design reviews, intermediate inspections, and final acceptance inspections, and to mark completed operation steps through the AI ​​system, verify whether key steps have not been omitted or tampered with, and record the completion of operations through the blockchain.

[0063] The deep fake detection technology includes the synchronization detection of audio and video by comparing the Mel frequency cepstral coefficients of the audio and the optical flow features of the lip movement in the video. The synchronization measurement is as follows: Audio Features and video features Matching is performed through dynamic time warping, and the matching degree is: in, is the Euclidean distance between feature vectors; N is the number of time points in the audio or video sequence; is the audio and video synchronization matching score; is the audio feature at the i-th time point; is the video optical flow feature at the i-th time point; if Less than threshold , the audio and video are considered out of sync and are judged to be forged.

[0064] Extract facial expression features from video frames and represent them as feature vectors , the calculation formula is: in, is the facial expression feature of the video at time t; Q is the feature dimension of the input video frame, including pixels and colors; is the input feature value of the video frame at a specific position and time t; is the weight coefficient in the facial feature extraction network, which represents the weight between the i-th feature point and the j-th input feature; is the number of facial region feature points, is the bias term of facial landmarks.

[0065] Extract the emotional features of the tone from the audio and represent them as feature vectors , the calculation formula is: Where S is the number of audio frames, Indicates an audio signal.

[0066] By calculating the correlation between facial expressions and tone of voice emotional features, the matching degree is defined as: in, is the facial expression feature of the video at time t, is the audio tone and emotional feature at time t, and are the means of facial expression and tone of voice emotion features, Indicates the matching degree between facial expression and tone of voice; if Less than threshold , then there is forgery.

[0067] By calculating the abnormal points of the spectrum, the audio forgery traces are detected and the spectrum change amount is defined for: Detect abnormal changes in the spectrum and define the anomaly score as: in, is the audio spectrum anomaly score; is the average value of the spectrum variation at frequency f; represents the frequency spectrum of the audio signal at frequency f before time t; if Exceeding the threshold , the audio has been fabricated or edited.

[0068] The brightness distribution of a video frame is defined as the sum of the pixel brightness. The brightness changes are analyzed using Laplace transform by detecting the spatial and temporal consistency of the brightness changes: in, Indicates the brightness change of the i-th frame; represents the brightness distribution of video frame j at time t; C represents the number of frames; Indicates the consistency score of light and shadow changes; represents the second-order derivative of the brightness of the j-th frame of the video in space; if the difference in light and shadow changes Greater than threshold , then there is forgery.

[0069] When deep fake detection technology identifies forged or tampered audio or video data, the system automatically triggers an alarm mechanism, records the information to the blockchain and notifies senior management, while generating a complete event report for auditing and tracing.

[0070] Embodiment 2 is another embodiment of the present invention, which provides a power marketing service behavior supervision and control system, referring to Figure 2The system includes: through multimodal data fusion technology, real-time collection of audio, video and location information of account managers during power access services; the system uses a data acquisition module to obtain multi-source data of account managers, including voice conversations, on-site images and GPS location information; the audio processing module converts audio signals into text and performs semantic analysis to detect sensitive words and inappropriate language; the video processing module uses behavior recognition technology to detect the work flow and dress specifications of account managers; the location matching module verifies whether the account manager's work location complies with regulations through GPS positioning.

[0071] The system performs deep forgery detection through the analysis module. The audio and video synchronization detection module detects the synchronization of audio and video features by matching the time series of the two. The facial expression matching detection module compares the facial expressions in the video with the tone and emotional features in the audio to detect whether they are consistent. The video light and shadow consistency detection module determines whether the video has been edited by analyzing the brightness changes of the video frames. The audio waveform analysis module detects traces of forgery in the audio by extracting spectral features. The detection results of each module are comprehensively calculated through the multimodal data fusion module to generate a forgery detection score.

[0072] The blockchain storage and verification module encrypts all collected and analyzed data and stores them in the blockchain, and the intelligent labeling and tracking module labels and tracks the key operation steps in the service process.

[0073] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

[0074] Embodiment 3, the third embodiment of the present invention, is different from the first two embodiments in that: If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0075] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0076] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0077] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0078] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0079] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for supervising and controlling power marketing service behavior, characterized in that: include, Collect audio, video and location information of the account manager during the power access service process, and perform multimodal data fusion analysis on the collected audio, video and location information to comprehensively determine whether the account manager's service behavior is compliant; Use AI image recognition technology to monitor the account manager's clothing in real time and determine whether the behavior is correct based on the fusion data; Based on blockchain technology, the account manager's behavior data in each service link is encrypted and stored and recorded in an unalterable manner. At the same time, the key operation steps in the service process are intelligently marked and tracked to accurately manage the account manager's operation process; The authenticity of the audio and video data provided by the account manager is verified through deep fake detection technology, and the final verification results are output.

2. A method for supervising and controlling power marketing service behavior according to claim 1, characterized in that: The multimodal data fusion analysis is to achieve a comprehensive assessment of the customer manager's behavior through a comprehensive analysis of audio, video and location information, including: Perform speech recognition on audio data and convert speech into text; Use natural language processing technology to perform semantic analysis on the converted text to identify whether the account manager has violated service regulations by using sensitive words or inappropriate language; Perform image recognition on video data and combine it with location information to analyze whether the account manager performs on-site operations correctly and verify whether he or she provides services at the specified location.

3. A method for supervising and controlling power marketing service behavior according to claim 2, characterized in that: The speech recognition of the audio data includes performing short-time Fourier transform on the collected audio signal and converting it into text representation using MFCC feature extraction: Get the Mel frequency cepstral coefficient features through the Mel filter: in, Represents the frequency spectrum of the audio signal at frequency f and time t; represents the time variable; f represents the frequency; t represents the time; represents the MFCC features at time t; M represents the number of Mel filters; Indicates the number of frequency band divisions; represents the frequency component; Represents the response value of the mth Mel filter; represents the integral representation of the audio signal; Represents the power spectrum of a frequency domain signal; represents the complex exponential function, j represents the imaginary unit; Convert audio features to text, i.e.: Where T is the text representation of the audio signal, and SR() represents the speech recognition function; Through predefined vocabulary sets or SVM based on sentiment classification, we can identify whether there are sensitive words or inappropriate language. The detection results are passed through the classifier function Output: in, Indicates the result of audio violation detection, 0 indicates no violation, and 1 indicates a violation.

4. A method for supervising and controlling power marketing service behavior according to claim 3, characterized in that: The image recognition of the video data includes inputting a video frame sequence and extracting behavior features from the video frames using an architecture combining a convolutional neural network and a long short-term memory network; For video signals, the optical flow features of lip motion are extracted by optical flow algorithm, which is expressed as: in, Is the video in position and the pixel value at time t, indicating the speed of lip movement; Represents the optical flow features extracted from the video data at time t; The convolutional neural network is used to identify the behavior of the video motion features. The convolutional neural network extracts the spatial and temporal features in the video frame to detect whether the customer manager's behavior is compliant: in, Indicates the result of video behavior recognition, 0 indicates compliant behavior, and 1 indicates illegal behavior; Represents the convolutional neural network model used to classify video features.

5. A method for supervising and controlling power marketing service behavior according to claim 4, characterized in that: The comprehensive judgment of whether the customer manager's service behavior is compliant includes: Semantic analysis is used in combination with image recognition to simultaneously analyze audio, video and location information, and when inconsistent behavior is detected, corresponding feedback information is immediately generated to remind the account manager to adjust the behavior; Through GPS positioning technology, the location information of the account manager is collected and expressed as the coordinate points of the time series: By calculating the preset location With actual location The Euclidean distance of: in, Indicates the preset location With actual location The Euclidean distance of , , They respectively represent the actual location information of the account manager at time t in spatial coordinates; , , They respectively represent the preset position information of the account manager at time t in the spatial coordinates; like Less than a predetermined threshold , the account manager is considered to be at the specified location; the location matching flag is specifically represented by: The violation detection results of audio processing , video behavior recognition results And location information matching results Conduct multi-modal fusion to comprehensively evaluate whether the account manager's behavior is compliant; Multimodal fusion obtains the final judgment result by weighted summing the classification results of audio, video and location information. It is expressed as: in, They represent the weight coefficients of audio violation detection results, video behavior recognition results, and location information matching results, respectively. The weight setting is optimized through training data. like Exceeding the preset threshold , then the account manager’s behavior is judged to be non-compliant.

6. A method for supervising and controlling power marketing service behavior according to claim 5, characterized in that: The key operation steps include on-site investigation, design review, intermediate testing and final acceptance; The intelligent labeling and tracking technology is used to automatically identify the service operations of customer managers during on-site surveys, design reviews, intermediate inspections, and final acceptance inspections, and to mark completed operation steps through the AI ​​system, verify whether key steps have not been omitted or tampered with, and record the completion of operations through the blockchain.

7. A method for supervising and controlling power marketing service behavior according to claim 6, characterized in that: The deep fake detection technology includes the synchronization detection of audio and video by comparing the Mel frequency cepstral coefficients of the audio and the optical flow features of the lip movement in the video. The synchronization measurement is as follows: Audio Features and video features Matching is performed through dynamic time warping, and the matching degree is: in, is the Euclidean distance between feature vectors; N is the number of time points in the audio or video sequence; is the audio and video synchronization matching score; is the audio feature at the i-th time point; is the video optical flow feature at the i-th time point; if Less than threshold , the audio and video are considered out of sync and are judged to be forged; By calculating the correlation between facial expressions and tone of voice emotional features, the matching degree is defined as: in, is the facial expression feature of the video at time t, is the audio tone and emotional feature at time t, and are the means of facial expression and tone of voice emotion features, Indicates the matching degree between facial expression and tone of voice; if Less than threshold , then there is forgery; By calculating the abnormal points of the spectrum, the audio forgery traces are detected and the spectrum change amount is defined for: Detect abnormal changes in the spectrum and define the anomaly score as: in, is the audio spectrum anomaly score; is the average value of the spectrum variation at frequency f; represents the frequency spectrum of the audio signal at frequency f before time t; if Exceeding the threshold , the audio has been fabricated or edited; The brightness distribution of a video frame is defined as the sum of the pixel brightness. The brightness changes are analyzed using Laplace transform by detecting the spatial and temporal consistency of the brightness changes: in, Indicates the brightness change of the i-th frame; represents the brightness distribution of video frame j at time t; C represents the number of frames; Indicates the consistency score of light and shadow changes; represents the second-order derivative of the brightness of the j-th frame of the video in space; if the difference in light and shadow changes Greater than threshold , then there is forgery; When deep fake detection technology identifies forged or tampered audio or video data, the system automatically triggers an alarm mechanism, records the information to the blockchain and notifies senior management, while generating a complete event report for auditing and tracing.

8. A system using a method for supervising and controlling power marketing service behavior as claimed in any one of claims 1 to 7, characterized in that: The system includes, through multimodal data fusion technology, real-time collection of audio, video and location information of account managers during the power access service process; the system uses the data acquisition module to obtain multi-source data of account managers, including voice conversations, on-site images and GPS location information; the audio processing module converts audio signals into text and performs semantic analysis to detect sensitive words and inappropriate language; the video processing module uses behavior recognition technology to detect the work process and dress specifications of account managers; the location matching module verifies whether the work location of the account manager complies with regulations through GPS positioning; The system uses the analysis module to detect deep fakes. The audio and video synchronization detection module detects the synchronization of the audio and video features by matching the time series of the two. The facial expression matching detection module compares the facial expressions in the video with the emotional tone in the audio to detect whether they are consistent. The video light and shadow consistency detection module determines whether the video has been edited by analyzing the brightness changes of the video frames. The audio waveform analysis module detects forgery traces in the audio by extracting spectrum features; the detection results of each module are comprehensively calculated by the multimodal data fusion module to generate a forgery detection score; The blockchain storage and verification module encrypts all collected and analyzed data and stores them in the blockchain, and the intelligent labeling and tracking module labels and tracks the key operation steps in the service process.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images