Text and audio and video fusion behavior recognition method, equipment and product
通过无线耳麦和网络摄像头获取数据,结合YOLOv5和语音识别网络模型进行模糊匹配,解决了工作状态下语音语调差异性导致的误判问题,提高了文本及音频视频融合行为识别的准确性和跨媒体 Automatic judgment ability of behavior recognition.
Patent Information
- Application Number
- CN202410989534.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-23
- Publication Date
- 2024-11-29
AI Technical Summary
The existing methods of fusion behavior recognition of text and audio and video fusion behaviors in the working state lead to misjudgment, affecting the accuracy of recognition.
Use wireless headsets and webcams to obtain audio and video data, use the trained YOLOv5 model to detect target operation effects, and combine the speech recognition network model to fuzzy match speech and passwords through Chinese character-pinyin dictionary mapping and fuzzy mapping rules, dynamic programming Method determines the maximum subsequence length to determine whether the speech file contains a password.
It improves the accuracy of cross-media behavior recognition, solves the recognition problem caused by the differences in accents of different personnel, and realizes the automated judgment of the operation of complex equipment whether it is carried out in accordance with the specifications.
Smart Images

Figure CN119046867A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of video image and speech recognition, and in particular to a text and audio / video fusion behavior recognition method, device and product. Background Art
[0002] With the rapid development of science and technology, video image and speech recognition technologies have been widely used in various fields, and cross-media fusion technology has become one of the important directions that has attracted much attention in the field of artificial intelligence and information technology. One of them is the compliance determination of personnel behavior. This technical field uses advanced technical means such as deep learning, computer vision and natural language processing to enable us to accurately identify and analyze personnel behavior in cross-media data, so as to determine whether it meets specific compliance standards.
[0003] Cross-media fusion technology requires efficient fusion and synchronization of different media data. The heterogeneity and complexity of various types of information make data integration challenging, and it is necessary to solve problems such as data format, standard and time sequence. In enterprise production management, the compliance determination of personnel operations is an important link to ensure the normal operation of the enterprise and employees' compliance with rules and regulations. Cross-media fusion technology can be applied to monitor the internal environment of the enterprise. By analyzing information such as employees' behavior, speech and communication relationships, the efficiency of compliance supervision can be improved and potential risks can be reduced. In the financial field, cross-media fusion technology can be used for risk management and fraud detection. By integrating images, videos, voice and text information, the system can more accurately identify suspicious behaviors, warn of potential fraud activities, and improve the security of financial institutions. With the popularity of social media, cross-media fusion technology can also be used for content supervision of social media. The system analyzes the images, videos and text content posted by users to determine whether there are violations, inappropriate remarks or sensitive information, helping social media platforms maintain a good community environment.
[0004] The text and audio-video fusion behavior recognition method is a product of the current era of informatization, automation, and high precision. It is also a very close issue in the fields of production management, security, and special equipment operation and maintenance management.
[0005] At present, there are many studies on text and audio and video fusion behavior recognition methods. For images and videos, commonly used feature extraction methods include convolutional neural networks (CNNs). For audio, speech recognition and audio feature extraction are the main means, while for text, natural language processing (NLP) technology is the key technical means. Among them, information fusion is the core of cross-media fusion. Common fusion methods include feature-level fusion, decision-level fusion, and model-level fusion. Through these methods, the system can better integrate and utilize information from different media. However, the above methods still have the following problems: For the speech recognition part, there is a certain difference between the individual's voice and intonation in the working state and Mandarin, which is easy to cause misjudgment.
[0006] Therefore, there is an urgent need to provide a new text standard and audio and video fusion behavior recognition method to solve the data recognition and fusion problems in cross-media behavior recognition and further improve the accuracy of data recognition and fusion. Summary of the invention
[0007] The purpose of the present invention is to provide a text and audio and video fusion behavior recognition method, device and product, which can solve the data recognition and fusion problems in cross-media behavior recognition and further improve the accuracy of data recognition and fusion.
[0008] To achieve the above object, the present invention provides the following solutions:
[0009] A text and audio / video fusion behavior recognition method, the method comprising:
[0010] The audio data and video data are obtained by using a wireless headset and a network camera; the camera is used to shoot the operating device to be monitored and collect the video data of the operating device to be monitored; the wireless headset is worn on the operator to obtain the audio data of the operator; the operating device includes: an operating panel and a screen;
[0011] Detect the target operation effect of the video data using the trained YOLOv5 model; and detect the command text sequence B and operation time T of the corresponding operation according to the target operation effect; the trained YOLOv5 model is used to monitor the target operation effect;
[0012] According to the audio data and the operation time T, the audio file of the set time before the operation occurs is intercepted;
[0013] The speech recognition network model is used to perform speech recognition on the audio file to obtain the sentence text sequence A; and the sentence text sequence A and the command text sequence B are fuzzy matched to obtain the matching status of the text and audio and video data.
[0014] Optionally, the trained YOLOv5 model is determined as follows:
[0015] Data annotation of operation panels and screens; the annotation content is the operation effects under all text specifications; the operation effects include: indicator lights lighting up, button status changes, and changes in designated icons on the screen;
[0016] Use the labeled data to train the YOLOv5 model to obtain a trained YOLOv5 model.
[0017] Optionally, the speech recognition network model includes: a wenet network.
[0018] Optionally, the speech recognition network model is used to perform speech recognition on the audio file to obtain a sentence text sequence A; and the sentence text sequence A and the command text sequence B are fuzzy matched to obtain the matching of the text and the audio and video data, specifically including:
[0019] Using the Chinese character-pinyin dictionary mapping, the sentence text sequence A and the command text sequence B are converted into the pinyin sequence a and the pinyin sequence b respectively;
[0020] Constructing fuzzy mapping rules between pinyin; the fuzzy mapping rules between pinyin include: mapping between retroflex and flat tongue sounds, and mapping between front and back nasals;
[0021] Perform fuzzy matching on the pinyin sequence b and the pinyin sequence a by using a dynamic programming method combined with fuzzy mapping rules to determine the maximum subsequence length of the pinyin sequence b in the pinyin sequence a;
[0022] Determine the similarity between the voice and the password according to the maximum subsequence length and the length of the pinyin sequence b;
[0023] Determine whether the voice file contains the command based on the similarity.
[0024] Optionally, the fuzzy matching of the pinyin sequence b and the pinyin sequence a using a dynamic programming method combined with a fuzzy mapping rule to determine the maximum subsequence length of the pinyin sequence b in the pinyin sequence a specifically includes:
[0025] Create a two-dimensional array dp according to the pinyin sequence b and the pinyin sequence a; initially, all elements are set to 0; the two-dimensional array dp is (len(a)+1)×(len(b)+1); len(a) is the length of the pinyin sequence a, and len(b) is the length of the pinyin sequence b;
[0026] Perform state transfer on the two-dimensional array dp;
[0027] Starting from dp[1][1], fill the dp array row by row and column by column;
[0028] Store the largest subsequence in dp[len(a)][len(b)].
[0029] A computer device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of a text and audio / video fusion behavior recognition method.
[0030] Optionally, the memory is a computer-readable storage medium.
[0031] A computer program product includes a computer program, which implements the steps of a text and audio-video fusion behavior recognition method when executed by a processor.
[0032] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:
[0033] The text and audio-video fusion behavior recognition method, device and product provided by the present invention can automatically judge whether the equipment operation is performed in accordance with the specification requirements through video and audio monitoring and AI recognition technology during the single-person or multi-person collaborative operation of complex equipment. The specification includes the command that needs to be issued before the operation, the operation that the operator needs to perform and the operation effect that will occur. It can solve the recognition problem caused by the differences in accents of different people, and solve the data recognition and fusion problems in cross-media behavior recognition. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0035] Figure 1 A flowchart of a text and audio / video fusion behavior recognition method provided by the present invention;
[0036] Figure 2 A diagram showing the deployment of cameras and wireless headsets. DETAILED DESCRIPTION
[0037] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0038] The purpose of the present invention is to provide a text and audio and video fusion behavior recognition method, device and product, which can solve the data recognition and fusion problems in cross-media behavior recognition and further improve the accuracy of data recognition and fusion.
[0039] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.
[0040] Example 1
[0041] like Figure 1 As shown, the present invention provides a text and audio and video fusion behavior recognition method, the method comprising:
[0042] S101, using a wireless headset and a network camera to obtain audio data and video data; the camera is used to shoot the operating device to be monitored and collect video data of the operating device to be monitored; the wireless headset is worn on the operator to obtain the audio data of the operator; the operating device includes: an operating panel and a screen;
[0043] like Figure 2 As shown, a network camera is installed to shoot the operating equipment that needs to be monitored, and the network camera, wireless headset receiver and server are connected via a network cable; wherein, the solid line is the network cable connection, and the dotted line is the wireless communication. During operation, all operators are required to wear wireless headsets, and the audio data and video data of the wireless headset and the network camera can be received in real time on the back-end server.
[0044] S102, using the trained YOLOv5 model to detect the target operation effect on the video data; and detecting the command text sequence B and operation time T of the corresponding operation according to the target operation effect; the trained YOLOv5 model is used to monitor the target operation effect;
[0045] Before S102, the method further includes: using OpenCV to connect to the real-time video streams of each camera and decoding the video streams frame by frame;
[0046] The trained YOLOv5 model determination process is as follows:
[0047] Data annotation is performed on the operation panel and screen; the annotation content is the operation effect under all text specifications; the operation effects include: indicator light lighting, button status change, and specified icon change on the screen; for example, after operation Q1 is completed, indicator light L1 on panel W1 becomes bright, or a specific icon T appears on the screen, then the annotation object is operation W1 or T.
[0048] The YOLOv5 model is trained using the labeled data to obtain a trained YOLOv5 model. The operation effect in the operation specification text is bound to the trigger effect to be detected, that is, if the specified operation effect is detected, it is considered that the specified operation has occurred.
[0049] S103, based on the audio data and the operation time T, an audio file of a set time before the operation occurs is intercepted; the set time is determined according to the time required for the operation, usually about one minute.
[0050] S104, using a speech recognition network model to perform speech recognition on the audio file to obtain a sentence text sequence A; and fuzzy matching is performed on the sentence text sequence A and the command text sequence B to obtain the matching status of the text and the audio and video data. The speech recognition network model includes but is not limited to: wenet network.
[0051] S104 specifically includes:
[0052] S401, using Chinese character-pinyin dictionary mapping, converting the sentence text sequence A and the command text sequence B into pinyin sequence a and pinyin sequence b respectively;
[0053] S401 specifically includes:
[0054] Using the Chinese character-pinyin dictionary mapping, the text sequence A is converted into a pinyin sequence, and the pinyins are separated by spaces to obtain a pinyin sequence a in the form of "ni hao";
[0055] Using the Chinese character-pinyin dictionary mapping, the command text sequence B is converted into a pinyin sequence, and the pinyins are separated by spaces to obtain a pinyin sequence b in the form of "ni hao";
[0056] S402, construct fuzzy mapping rules between pinyin; the fuzzy mapping rules between pinyin include: retroflex-flat tongue mapping, mapping between front and back nasals; specifically, retroflex-flat tongue mapping, that is, mapping between pinyin QhP and QP, where Q and P can be any pinyin strings, and h represents the character h itself. Similarly, mapping can be performed between front and back nasals, that is, mapping can be performed between pinyin Qg and Q, where Q is any pinyin string and g is the character itself. Furthermore, objects that are easily confused in pronunciation of speech sentences can be mapped and added, and the set of all mapping rules is F, in which the mapping rules are {f1,f2,f3,...}.
[0057] S403, fuzzy matching is performed on the pinyin sequence b and the pinyin sequence a using a dynamic programming method combined with fuzzy mapping rules to determine the maximum subsequence length of the pinyin sequence b in the pinyin sequence a; when the pinyin P1 meets the rules, it can be mapped to the pinyin P2 according to the rules.
[0058] S403 specifically includes:
[0059] S1, create a two-dimensional array dp according to the pinyin sequence b and the pinyin sequence a; initially, all elements are set to 0; the two-dimensional array dp is (len(a)+1)×(len(b)+1); len(a) is the length of the pinyin sequence a, and len(b) is the length of the pinyin sequence b;
[0060] S2, transfer the state of the two-dimensional array dp;
[0061] For each pair (i, j), consider the ith pinyin of a and the jth pinyin of b. If a[i-1] == b[j-1], it means that the ith element of a and the jth element of b are equal, then the current matching length is the previous matching length plus 1, that is:
[0062] dp[i][j]=dp[i-1][j-1]+1;
[0063] Different from the conventional dynamic programming maximum subsequence matching, here if there is a mapping rule f in the mapping rule set F, such that:
[0064] a[i-1] == f(b[j-1]);
[0065] Then it is also considered that a[i-1]==b[j-1] holds. Otherwise, it means that the i-th element of A and the j-th element of B are not equal, and the current matching length is the maximum value of the same column in the previous row and the same column in the previous row, that is:
[0066] dp[i][j]=max(dp[i-1][j],dp[i][j-1]);
[0067] S3, starting from dp[1][1], fill the dp array row by row and column by column; the value of each dp[i][j] depends on the three elements above its left, directly above it, and to its left.
[0068] S4, store the maximum subsequence in dp[len(a)][len(b)].
[0069] S404, determining the similarity between the voice and the password according to the maximum subsequence length and the length of the pinyin sequence b;
[0070] The ratio of the length of the largest subsequence dp[len(a)][len(b)] to the length of the pinyin sequence b len(b) determines the similarity between the speech and the password. If dp[len(a)][len(b)] divided by len(b) is greater than a threshold, such as 0.8, then the speech is considered to contain the password to be detected.
[0071] S405, judging whether the voice file contains a command according to the similarity.
[0072] Until the operation is completed or the video ends, the matching status of all detected action sequences and commands with specifications can be obtained.
[0073] Example 2
[0074] The present invention provides a computer device, which can be a database. The computer device includes a processor, a memory, an input / output interface (I / O for short) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store pending transactions. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a text and audio and video fusion behavior recognition method is implemented.
[0075] The memory is a computer-readable storage medium on which a computer program is stored. When the computer program is executed by the processor, the steps in the above-mentioned text and audio-video fusion behavior recognition method are implemented.
[0076] Example 3
[0077] The present invention also provides a computer program product, including a computer program, which implements the steps in the above-mentioned text and audio and video fusion behavior recognition method when executed by a processor.
[0078] It should be noted that the object information (including but not limited to object device information, object personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the object or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.
[0079] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited thereto.
[0080] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0081] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of the present invention. At the same time, for those skilled in the art, according to the idea of the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.
Claims
1. A text and audio / video fusion behavior recognition method, characterized in that: The method comprises: The audio data and video data are obtained by using a wireless headset and a network camera; the camera is used to shoot the operating device to be monitored and collect the video data of the operating device to be monitored; the wireless headset is worn on the operator to obtain the audio data of the operator; the operating device includes: an operating panel and a screen; Detect the target operation effect of the video data using the trained YOLOv5 model; and detect the command text sequence B and operation time T of the corresponding operation according to the target operation effect; the trained YOLOv5 model is used to monitor the target operation effect; According to the audio data and the operation time T, the audio file of the set time before the operation occurs is intercepted; The speech recognition network model is used to perform speech recognition on the audio file to obtain the sentence text sequence A; and the sentence text sequence A and the command text sequence B are fuzzy matched to obtain the matching status of the text and audio and video data.
2. The text and audio / video fusion behavior recognition method according to claim 1 is characterized in that: The trained YOLOv5 model determination process is as follows: Data annotation of operation panels and screens; the annotation content is the operation effects under all text specifications; the operation effects include: indicator lights lighting up, button status changes, and changes in designated icons on the screen; Use the labeled data to train the YOLOv5 model to obtain a trained YOLOv5 model.
3. The text and audio / video fusion behavior recognition method according to claim 1 is characterized in that: The speech recognition network model includes: a wenet network.
4. The text and audio / video fusion behavior recognition method according to claim 1 is characterized in that: The speech recognition network model is used to perform speech recognition on the audio file to obtain a sentence text sequence A; The sentence text sequence A and the command text sequence B are fuzzily matched to obtain the matching conditions of the text and audio and video data, including: Using the Chinese character-pinyin dictionary mapping, the sentence text sequence A and the command text sequence B are converted into the pinyin sequence a and the pinyin sequence b respectively; Constructing fuzzy mapping rules between pinyin; the fuzzy mapping rules between pinyin include: mapping between retroflex and flat tongue sounds, and mapping between front and back nasals; Perform fuzzy matching on the pinyin sequence b and the pinyin sequence a by using a dynamic programming method combined with fuzzy mapping rules to determine the maximum subsequence length of the pinyin sequence b in the pinyin sequence a; Determine the similarity between the voice and the password according to the maximum subsequence length and the length of the pinyin sequence b; Determine whether the voice file contains the command based on the similarity.
5. The text and audio / video fusion behavior recognition method according to claim 4 is characterized in that: The method of performing fuzzy matching on the pinyin sequence b and the pinyin sequence a by using a dynamic programming method combined with a fuzzy mapping rule to determine the maximum subsequence length of the pinyin sequence b in the pinyin sequence a specifically includes: Create a two-dimensional array dp according to the pinyin sequence b and the pinyin sequence a; initially, all elements are set to 0; the two-dimensional array dp is (len(a)+1)×(len(b)+1); len(a) is the length of the pinyin sequence a, and len(b) is the length of the pinyin sequence b; Perform state transfer on the two-dimensional array dp; Starting from dp[1][1], fill the dp array row by row and column by column; Store the largest subsequence in dp[len(a)][len(b)].
6. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the steps of a text and audio-video fusion behavior recognition method as described in any one of claims 1-5.
7. A computer device according to claim 6, characterized in that: The memory is a computer-readable storage medium.
8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of a text and audio / video fusion behavior recognition method described in any one of claims 1-5 are implemented.
