Autonomous video and audio enabling method based on semantic analysis

Abstract

The invention relates to the technical field of data analysis, in particular to a video and audio autonomous enabling method based on semantic analysis, which analyzes voice content in audio, combines scene information in a video, intelligently dispatches a video camera, and preferentially selects an area most relevant to a target for real-time analysis. According to the scheme, the requirement for real-time analysis of all camera video streams is effectively reduced, use of system resources is optimized, follow-up behaviors of a target are speculated through semantic analysis, and the response speed and positioning precision of a monitoring system are further improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data analysis, and in particular to an audio and video autonomous enabling method based on semantic analysis. Background Art

[0002] With the growing demand for video surveillance systems in smart cities and public security, especially in large-scale, multi-camera surveillance scenarios, how to quickly and accurately locate and track specific people has become a technical problem that needs to be solved. Traditional video surveillance systems usually rely on real-time analysis of video streams from all cameras, but in actual applications, the amount of computation required for video analysis is huge, especially when there are a large number of surveillance devices. Real-time analysis of all video streams will not only waste computing resources, but may also cause system delays, affecting the real-time performance and response efficiency of the surveillance system.

[0003] In addition, existing systems often ignore the semantic information of video content and rely only on simple processing of video streams, which cannot deeply understand the key information in the video, resulting in inaccurate positioning and behavior prediction of targets. Therefore, how to quickly identify and locate specific targets in massive video data and optimize the computational burden of real-time parsing by analyzing the semantic information in the video has become the key to improving system efficiency.

[0004] To this end, this application designs a method for autonomous audio and video empowerment based on semantic analysis, and solves the above-mentioned problems. Summary of the invention

[0005] In order to make up for the deficiencies in the prior art, the present invention provides an audio and video autonomous enabling method based on semantic analysis.

[0006] A method for autonomously enabling video and audio based on semantic analysis, characterized by comprising the following steps: S1, the audio-visual autonomous empowerment system based on semantic analysis obtains multi-source data and determines the target's historical trajectory, and conducts a preliminary analysis of the target's activity area and activity pattern; S2, sorting by historical trajectories, determining the priority search area; S3, real-time video and audio data analysis; S4, semantic analysis and behavior prediction; S5, result output and instruction execution.

[0007] Furthermore, in order to better realize the present invention, the S1 is specifically: S11, obtaining relevant information of the target from multiple data sources, including video surveillance data, audio data, smart device sensor data, and mobile phone card positioning information. This information will be used to construct the historical trajectory of the target; S12, on this basis, the system performs data preprocessing by analyzing the target's behavior history, including frequently visited places and frequently active areas. These historical trajectory information will be used to narrow the scope of real-time video analysis and avoid unnecessary global video analysis, thereby reducing computing pressure and improving efficiency.

[0008] Furthermore, in order to better implement the present invention, the video surveillance data provides a real-time visual image that can identify the appearance features and movement trajectory of the target; The audio data can confirm the existence of the target and its possible behavior through sound analysis, including conversation and environmental sound; The smart device sensor data, including access control systems and vehicle GPS, can further verify the location and activity status of the target; The mobile phone card positioning information can help to accurately locate people in a large area.

[0009] Further, in order to better implement the present invention, S2 is specifically: S21, based on the historical trajectory data of the target, the system will prioritize the cameras in the monitoring area; S22, the priority in the historical trajectory is determined by the target’s activity frequency, residence time, and the importance of the places visited. For example, if the target frequently enters and exits a specific area, including shopping malls and office buildings, the cameras in that area will be given a higher priority; conversely, areas that the target rarely passes through will have a lower priority; S23, the system automatically selects the most suitable camera for real-time monitoring based on these priorities.

[0010] Further, in order to better implement the present invention, S3 is specifically: S31, in the priority camera, the system starts to analyze the video and audio data in real time; video analysis includes target detection, tracking and recognition; S32, through computer vision technology, the system can detect possible targets from real-time video streams and track their movement trajectories; at this time, the target's facial recognition, clothing features, and body movement information will be extracted to ensure accurate target identification; S33, at the same time, the analysis of audio data is also carried out simultaneously. Through the sound recognition technology, the system can extract the conversation content, environmental sound and possible command information in the audio. For example, if the conversation content or environmental sound related to the target is detected in the audio, including "run" or "help", the system will obtain these key information in time and combine them with the video data to further confirm the behavior of the target; S34, at this stage, the system will also use voice recognition technology to identify voice commands in the video and determine whether there is instruction information related to the target; if there is instruction content, such as "arrive at location A", it will further track the target's position changes.

[0011] Further, in order to better implement the present invention, the S4 is specifically: S41, through semantic analysis technology, deeply analyzes the key information in the video and audio. Semantic analysis can understand the specific meaning of the video and audio content. For example, by analyzing the target's conversation content, such as "I want to go to the library", the system can infer the target's next action; S42, the system can also combine the deep learning model to predict the target's behavior pattern, so as to more accurately determine the target's possible destination.

[0012] Furthermore, in order to better implement the present invention, the semantic analysis is not limited to simple command recognition, but also includes multi-dimensional analysis of environmental sounds, tone, and situational factors; for example, if the system detects that the target says "I want to leave" in a noisy environment, the system can recognize the sentence and judge whether the target intends to leave the area in combination with the surrounding environment. The core of this step is to use natural language processing and semantic reasoning algorithms to predict the next step of behavior based on the target's historical behavior and current voice / video information.

[0013] Furthermore, in order to better realize the present invention, the S5 is specifically as follows: through real-time video and audio data analysis, the system generates prediction results about the target behavior and transmits them to relevant equipment for execution; for example, if the target is detected approaching a dangerous area or a specific location, the system can automatically trigger an alarm or issue a command to notify security personnel to further intervene; if the target's voice or behavior analysis shows that it is about to perform a specific behavior, such as leaving a certain area or entering a certain building, the system can issue a command to the relevant equipment to automatically start monitoring or other necessary response measures.

[0014] The beneficial effects of the present invention are: Compared with the prior art, the present invention has the following advantages: First, by integrating multi-source information such as audio and video, smart devices and mobile phone positioning to obtain the historical trajectory of the target, and scheduling the camera according to the trajectory priority, the real-time processing of all video sources is avoided, the computational burden is greatly reduced, and the efficiency is improved. Secondly, the accuracy of target recognition is enhanced by combining audio and video data with speech recognition. The conversation content and ambient sound information in the audio provide more context for video analysis, improving the accuracy of target positioning and behavior analysis. Third, the present invention introduces semantic analysis technology, which can predict the next behavior of the target, and combines video and audio information to provide intelligent behavior prediction and decision support. Finally, the present invention can automatically schedule related equipment based on the prediction of target behavior through an intelligent instruction execution mechanism, thereby improving the response speed and accuracy of smart city command and dispatch. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0016] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Generally, the components of the embodiments of the present invention described and shown in the accompanying drawings here can be arranged and designed in various different configurations.

[0017] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present invention.

[0018] Figure 1 This is a specific embodiment of the present invention, which is a method and system for autonomous audio and video empowerment based on semantic analysis.

[0019] In the prior art, common video surveillance solutions include monitoring systems based on real-time video analysis and behavior recognition based on target detection. These systems track targets by analyzing objects or people in the video, but when faced with a large number of video streams, they often cannot effectively reduce the computing burden and have the problem of response delay. In addition, some systems use speech recognition and natural language processing (NLP) technology to analyze conversations or voice commands in the video, but these methods usually process video and audio data separately and lack the ability to perform audio and video fusion analysis, resulting in the inability to accurately infer target behavior or optimize resource scheduling in complex scenarios. Therefore, the prior art still has certain limitations in improving real-time performance, reducing computing pressure, and accurately predicting target behavior.

[0020] Based on the existing technology, this embodiment realizes more intelligent and efficient target positioning and behavior prediction by combining semantic analysis of video and audio data. Different from the existing solutions, the present invention not only uses the target detection and behavior recognition technology in the video, but also conducts comprehensive analysis of the video and audio content to deeply understand the voice information and scenes in the video, thereby reducing the computational burden while improving the accuracy of target tracking and behavior prediction.

[0021] The specific method of this embodiment is as follows: Step 1: Obtain multi-source data and determine the target historical trajectory First, relevant information about the target is obtained from multiple data sources, including video surveillance data, audio data, smart device sensor data, and mobile phone card positioning information. This information will be used to construct the historical trajectory of the target. Video surveillance data provides real-time visual images that can identify the appearance characteristics and movement trajectory of the target; audio data can confirm the existence of the target and its possible behavior (for example, conversation, ambient sound, etc.) through sound analysis; smart device data (such as access control systems, vehicle-mounted GPS, etc.) can further verify the location and activity status of the target; mobile phone card positioning information can help accurately locate people in a large area. Through the integration of this information, the system can generate the historical trajectory data of the target and conduct a preliminary analysis of the target's activity area and activity pattern.

[0022] On this basis, the system can perform data preprocessing by analyzing the target's behavioral history (such as frequently visited places, frequently active areas, etc.). This historical trajectory information will be used to narrow the scope of real-time video analysis and avoid unnecessary global video analysis, thereby reducing computing pressure and improving efficiency.

[0023] Step 2: Sort by historical tracks and determine the priority search area Based on the historical trajectory data of the target, the system will prioritize the cameras in the monitoring area. The priority in the historical trajectory is usually determined by the target's activity frequency, residence time, and the importance of the places visited. For example, if the target frequently enters and exits a specific area (such as a shopping mall, office building, etc.), the camera in that area will be given a higher priority; conversely, areas where the target rarely passes will have a lower priority. The system will automatically select the most suitable camera for real-time monitoring based on these priorities.

[0024] The goal of this step is to reduce the pressure of real-time video analysis by prioritizing areas with more frequent target activities, while improving the efficiency and accuracy of target search. If a target is found in a high-priority camera area, the system will immediately start parsing the video and audio data of the area and predicting the target's behavior.

[0025] Step 3: Real-time video and audio data analysis In the priority camera, the system starts to analyze the video and audio data in real time. Video analysis includes steps such as target detection, tracking and recognition. Through computer vision technology, the system can detect possible targets from the real-time video stream and track their movement trajectory. At this time, the target's facial recognition, clothing features, body movements and other information will be extracted to ensure accurate target recognition.

[0026] At the same time, audio data analysis is also carried out. Through sound recognition technology, the system can extract the conversation content, ambient sound and possible command information in the audio. For example, if the audio detects conversation content or ambient sound related to the target (such as "run", "help", etc.), the system will obtain this key information in time and combine it with the video data to further confirm the target's behavior.

[0027] At this stage, the system will also use voice recognition technology to identify voice commands in the video and determine whether there is any instruction information related to the target. If there is instruction content (such as "reach location A"), it will further track the target's location changes.

[0028] Step 4: Semantic analysis and behavior prediction Through semantic analysis technology, the key information in the video and audio is deeply analyzed. Semantic analysis can understand the specific meaning of the video and audio content. For example, by analyzing the target's conversation content (such as "I want to go to the library"), the system can infer the target's next move. The system can also combine deep learning models to predict the target's behavior pattern, so as to more accurately determine the target's possible destination.

[0029] Semantic analysis is not limited to simple command recognition, but also includes multi-dimensional analysis of factors such as ambient sound, tone, and context. For example, if the system detects that the target says "I want to leave" in a noisy environment, the system can recognize the sentence and determine whether the target intends to leave the area based on the surrounding environment. The core of this step is to use natural language processing (NLP) and semantic reasoning algorithms to predict the next step of behavior based on the target's historical behavior and current voice / video information.

[0030] Step 5: Result output and instruction execution Through real-time video and audio data analysis, the system generates predictions about the target's behavior and transmits them to related devices for execution. For example, if the target is detected approaching a dangerous area or a specific location, the system can automatically trigger an alarm or issue a command to notify security personnel for further intervention. If the target's voice or behavior analysis shows that it is about to perform a specific behavior (such as leaving an area or entering a building), the system can issue a command to the relevant equipment to automatically start monitoring or other necessary response measures.

[0031] Through this step, the system can not only monitor the behavior of the target in real time, but also respond quickly according to the predicted results. This real-time response capability is an important part of the smart city command and dispatch system, which can effectively improve the efficiency of urban management and the ability to respond to emergencies.

[0032] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Other modifications or equivalent substitutions made to the technical solution of the present invention by ordinary technicians in the field should be included in the scope of the claims of the present invention as long as they do not depart from the spirit and scope of the technical solution of the present invention.

Claims

1. A method for autonomous audio and video empowerment based on semantic analysis, characterized in that: The following steps are involved: S1, the audio-visual autonomous empowerment system based on semantic analysis obtains multi-source data and determines the target's historical trajectory, and conducts a preliminary analysis of the target's activity area and activity pattern; S2, sorting by historical trajectories, determining the priority search area; S3, real-time video and audio data analysis; S4, semantic analysis and behavior prediction; S5, result output and instruction execution.

2. The method for autonomous audio and video empowerment based on semantic analysis according to claim 1 is characterized in that: The S1 is specifically: S11, obtaining relevant information of the target from multiple data sources, including video surveillance data, audio data, smart device sensor data, and mobile phone card positioning information. This information will be used to construct the historical trajectory of the target; S12, on this basis, the system performs data preprocessing by analyzing the target's behavioral history, including frequently visited places and frequent activity areas. These historical trajectory information will be used to narrow the scope of real-time video analysis and avoid unnecessary global video analysis, thereby reducing computing pressure and improving efficiency.

3. The method for autonomous audio and video empowerment based on semantic analysis according to claim 2 is characterized in that: The video surveillance data provides real-time visual images, which can identify the appearance features and movement trajectory of the target; The audio data can confirm the existence of the target and its possible behavior through sound analysis, including conversation and environmental sound; The smart device sensor data, including access control systems and vehicle GPS, can further verify the location and activity status of the target; The mobile phone card positioning information can help to accurately locate people in a large area.

4. The method for autonomous audio and video empowerment based on semantic analysis according to claim 1 is characterized in that: The S2 is specifically: S21, based on the historical trajectory data of the target, the system will prioritize the cameras in the monitoring area; S22, the priority in the historical trajectory is determined by the target’s activity frequency, residence time, and the importance of the places visited. For example, if the target frequently enters and exits a specific area, including shopping malls and office buildings, the cameras in that area will be given a higher priority; On the contrary, the area where the target passes less frequently has a lower priority; S23, the system automatically selects the most suitable camera for real-time monitoring based on these priorities.

5. The method for autonomous audio and video empowerment based on semantic analysis according to claim 1 is characterized in that: The S3 is specifically: S31, in the priority camera, the system starts to analyze the video and audio data in real time; video analysis includes target detection, tracking and recognition; S32, through computer vision technology, the system can detect possible targets from real-time video streams and track their movement trajectories; at this time, the target's facial recognition, clothing features, and body movement information will be extracted to ensure accurate target identification; S33, at the same time, the analysis of audio data is also carried out simultaneously. Through the sound recognition technology, the system can extract the conversation content, environmental sound and possible command information in the audio. For example, if the conversation content or environmental sound related to the target is detected in the audio, including "run" and "help", the system will obtain these key information in time and combine them with the video data to further confirm the behavior of the target; S34, at this stage, the system will also use voice recognition technology to identify voice commands in the video and determine whether there is instruction information related to the target; if there is instruction content, such as "arrive at location A", it will further track the target's position changes.

6. The method for autonomous audio and video empowerment based on semantic analysis according to claim 1 is characterized in that: The S4 is specifically: S41, through semantic analysis technology, deeply analyzes the key information in the video and audio. Semantic analysis can understand the specific meaning of the video and audio content. For example, by analyzing the target's conversation content, such as "I want to go to the library", the system can infer the target's next action; S42, the system can also combine the deep learning model to predict the target's behavior pattern, so as to more accurately determine the target's possible destination.

7. The method for autonomous audio and video empowerment based on semantic analysis according to claim 6 is characterized in that: The semantic analysis is not limited to simple command recognition, but also includes multi-dimensional analysis of environmental sounds, tone, and situational factors. For example, if the system detects that the target says "I want to leave" in a noisy environment, the system can recognize the sentence and judge whether the target intends to leave the area based on the surrounding environment. The core of this step is to use natural language processing and semantic reasoning algorithms to predict the next step of behavior based on the target's historical behavior and current voice / video information.

8. The method for autonomous audio and video empowerment based on semantic analysis according to claim 1, characterized in that: Specifically, S5 is as follows: through real-time video and audio data analysis, the system generates prediction results about the target behavior and transmits them to relevant equipment for execution; for example, if the target is detected approaching a dangerous area or a specific location, the system can automatically trigger an alarm or issue a command to notify security personnel to further intervene; if the target's voice or behavior analysis shows that it is about to perform a specific behavior, such as leaving a certain area or entering a certain building, the system can issue a command to the relevant equipment to automatically start monitoring or other necessary response measures.

Images