Scene recognition method, system and storage medium based on speech classification

Abstract

The present invention relates to the technical field of security robots, and in particular to a scene recognition method, system and storage medium based on voice classification, wherein the system comprises a robot, and the robot comprises the following modules: a sound detection module: used to detect surrounding sounds and locate the sound source when the sound is detected; a mobile control module: used to control the robot's travel direction according to the result data of the sound source location; a sound classification module: used to input the acquired audio data into a trained neural network classification model for recognition and classification during the robot's continuous movement, wherein the recognition and classification results include artificially made sounds and environmental sounds; and a data analysis module: used to analyze the number of people and the purpose of the action in the current geographical environment according to the recognition and classification results. The present invention can accurately and efficiently complete patrols, and solves the problem of blind spots in existing manual patrols and video surveillance.

Description

Technical Field

[0001] The present invention relates to the technical field of security robots, and in particular to a scene recognition method, system and storage medium based on speech classification. Background Art

[0002] With the continuous improvement of social living standards, the number and size of stores, warehouses, factories and other places are increasing, so the security patrol tasks of the corresponding places are becoming increasingly heavy. At present, security personnel patrol and fixed-position surveillance cameras are basically used to achieve security monitoring tasks in such places. However, during the night time, security personnel are prone to fatigue, and there are problems such as reduced patrol sharpness and lack of concentration. Fixed camera monitoring has disadvantages such as blind spots. As a result, security personnel often have no time to issue an alarm when encountering emergencies, which is prone to problems. Summary of the invention

[0003] One of the objectives of the present invention is to provide a scene recognition system based on speech classification to solve the above-mentioned problems.

[0004] In order to achieve the above object, a scene recognition system based on speech classification is provided, including a robot, wherein the robot includes the following modules:

[0005] Sound detection module: used to detect surrounding sounds and locate the sound source when sound is detected;

[0006] Mobile control module: used to control the robot's moving direction according to the result data of sound source positioning;

[0007] Sound classification module: used to input the acquired audio data into the trained neural network classification model for recognition and classification during the robot's continuous movement. The recognition and classification results include artificial sounds and environmental sounds.

[0008] Data analysis module: used to analyze the number of people and purpose of actions in the current geographical environment based on the results of identification and classification.

[0009] Principles and advantages:

[0010] 1. The robot is set up with the sound detection module and the mobile control module. When the robot detects the sound, it can locate the sound source according to the sound, so that the robot can find the location of the existing situation. Compared with security personnel and fixed cameras, it can patrol freely without blind spots, and will not feel sleepy in the quiet environment at night. It has higher accuracy and can find the location in time when encountering emergencies, and go for patrol and investigation.

[0011] 2. The settings of the sound classification module and the data analysis module will input the acquired audio data into the trained neural network classification model for identification and classification, and use the existing and very mature neural network technology to quickly and accurately classify the sounds. For example, it can classify the footsteps, talking, knocking, glass breaking, object knocking, etc. in the current environmental sounds, and then hand them over to the data analysis module for subsequent analysis to analyze what problems will arise from the current phenomenon, so as to facilitate timely alarm to avoid the occurrence of problems.

[0012] Furthermore, the following modules are also included: a path planning module: used to obtain environmental data through the robot's laser radar, and re-plan the robot's travel direction based on the result data of sound source positioning;

[0013] Light control module: used to turn off the robot's lighting equipment when the robot moves according to the direction of travel.

[0014] The use of laser radar in the path planning module is not only to facilitate the robot to re-plan its travel direction in combination with the result data of sound source positioning, but also to prevent the robot from bumping into other objects during its travel, causing damage and noise, thereby disturbing intruders. The lighting control module turns off the robot's lighting equipment when the robot moves in the direction of travel to avoid disturbing intruders. This makes it easier for the robot to get closer to the intruders to obtain more accurate sound data, thereby ensuring the accuracy of data analysis.

[0015] Furthermore, the sound classification module is also used to pre-process the acquired sound, and specifically includes the following sub-modules:

[0016] Silence removal submodule: used to remove the silence part at the beginning and end of the acquired audio data;

[0017] Sound framing submodule: used to perform framing processing on the audio data after silence removal through a moving window function, and to make the frames overlap by a specified value;

[0018] Feature extraction submodule: used to extract features from the framed audio data.

[0019] By preprocessing the acquired sound, it is convenient to improve the working efficiency of the neural network model and enhance the classification accuracy.

[0020] Furthermore, the data analysis module specifically includes the following submodules:

[0021] Semantic analysis submodule: used to perform speech recognition on the acquired audio data, and perform semantic analysis based on the result of speech recognition to obtain the first action purpose;

[0022] Behavior analysis submodule: used to infer and analyze the second action purpose based on the artificially made sound, and to conduct a comprehensive analysis based on the first action purpose and the second action purpose to obtain the action target;

[0023] The number analysis submodule is used to perform a tone analysis on the voices in the man-made sounds to obtain the first number of people; it is also used to perform a tone analysis on the footsteps in the man-made sounds to obtain the second number of people; it is also used to analyze the number range data based on the first number of people and the second number of people.

[0024] The semantic analysis submodule and the behavior analysis submodule make it easy to understand the action targets of the intruders, so as to issue an alarm in time or take relevant measures to avoid the problem. The number analysis submodule is set up to analyze the number of intruders, so as to arrange the number of personnel to drive them away.

[0025] A second object of the present invention is to provide a scene recognition method based on speech classification, which is applied to the above system and specifically comprises the following steps:

[0026] Sound detection step: The robot detects surrounding sounds and locates the sound source when a sound is detected;

[0027] Mobile control step: Control the robot's moving direction according to the result data of sound source positioning;

[0028] Sound classification step: while the robot is moving, the acquired audio data is input into a trained neural network classification model for recognition and classification, wherein the recognition and classification results include artificially created sounds and environmental sounds;

[0029] Data analysis steps: Analyze the number of people and purpose of actions in the current geographical environment based on the results of identification and classification.

[0030] Principles and advantages:

[0031] 1. The sound detection step and the mobile control step are set together. When the robot detects the sound, it can locate the sound source according to the sound, so that the robot can find the location of the existing situation. Compared with security personnel and fixed cameras, it can patrol freely without blind spots, and will not feel sleepy in the quiet environment at night. It has higher accuracy and can find the location in time when encountering an emergency, and go to patrol and investigate.

[0032] 2. The sound classification step and the data analysis step are set up to input the acquired audio data into the trained neural network classification model for identification and classification, and use the existing and very mature neural network technology to quickly and accurately classify the sound. For example, it can classify the footsteps, talking, knocking, glass breaking, objects knocking, etc. in the current environment sound, and then hand it over to the data analysis step for subsequent analysis to analyze what problems the current phenomenon will cause, so as to facilitate timely alarm to avoid the occurrence of problems.

[0033] Further, the method further comprises the following steps:

[0034] Path planning steps: Obtain environmental data through the robot's laser radar, and re-plan the robot's travel direction based on the result data of sound source positioning;

[0035] Light control steps: When the robot moves according to the direction of travel, turn off the robot's lighting equipment.

[0036] The use of laser radar in the path planning step is not only to facilitate the robot to re-plan its travel direction in combination with the result data of sound source positioning, but also to prevent the robot from bumping into other objects during its travel, causing damage and noise, thereby disturbing intruders. The lighting control step is to turn off the robot's lighting equipment when the robot moves in the direction of travel to avoid disturbing intruders. This makes it easier for the robot to get closer to the intruders to obtain more accurate sound data, thereby ensuring the accuracy of data analysis.

[0037] Furthermore, the sound classification step is also used to pre-process the acquired sound, and specifically includes the following sub-steps:

[0038] Silence removal sub-step: removing the silence parts at the beginning and end of the acquired audio data;

[0039] Sound framing sub-step: framing the audio data after silence removal by moving the window function, and making the overlap between the frames have a specified value;

[0040] Feature extraction sub-step: extract features from the audio data that has completed framing.

[0041] By preprocessing the acquired sound, it is convenient to improve the working efficiency of the neural network model and enhance the classification accuracy.

[0042] Furthermore, the data analysis step specifically includes the following sub-steps:

[0043] Semantic analysis sub-step: perform speech recognition on the acquired audio data, and perform semantic analysis based on the results of speech recognition to obtain the purpose of the action;

[0044] Behavior analysis sub-step: inferring and analyzing the second action purpose based on the artificially made sound, and performing a comprehensive analysis based on the first action purpose and the second action purpose to obtain the action goal;

[0045] Number analysis sub-step: perform tone analysis on the speaking sound to obtain the first number of people; also perform tone analysis on the footsteps to obtain the second number of people; also perform analysis on the number of people range data based on the first number of people and the second number of people.

[0046] The semantic analysis sub-step and the behavior analysis sub-step make it easy to understand the action target of the intruder, so as to issue an alarm in time or take relevant measures to avoid the problem. The number analysis sub-step makes it easy to analyze the number of intruders, so as to arrange the number of personnel to drive them away.

[0047] A third object of the present invention is to provide a computer-readable storage medium, which includes a program for scene recognition based on speech classification. When the program for scene recognition based on speech classification is executed by a processor, the steps of the above-mentioned scene recognition method based on speech classification are implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 is a flowchart of a scene recognition system based on speech classification in Embodiment 1 of the present invention;

[0049] Figure 2 This is the network structure diagram of the speech classification model;

[0050] Figure 3 Flowchart of the process for speech classification. DETAILED DESCRIPTION

[0051] The following is further described in detail through specific implementation methods:

[0052] Embodiment 1

[0053] A scene recognition system based on speech classification, basically as shown in the attached Figure 1 As shown: including a robot, the hardware architecture of the robot includes a chassis, and a moving wheel is arranged under the chassis. The chassis is provided with a low-power AI computing motherboard, a laser radar, a sound source positioning module, a sound recognition sensor, a battery, a communication module, etc. The software architecture of the robot includes the following modules:

[0054] Sound detection module: used to detect surrounding sounds through sound recognition sensors, and locate the sound source through the sound source localization module when sounds are detected;

[0055] Mobile control module: used to control the robot's moving direction according to the result data of sound source positioning;

[0056] Path planning module: used to obtain environmental data through the robot's laser radar, and re-plan the robot's travel direction based on the result data of sound source positioning;

[0057] Light control module: used to turn off the robot's lighting equipment when the robot moves according to the direction of travel.

[0058] Sound classification module: used to input the acquired audio data into the trained neural network classification model (i.e., speech classification model) for recognition and classification during the robot's continuous movement; the recognition and classification results include man-made sounds and environmental sounds; the man-made sounds include footsteps, talking, knocking, glass breaking, and object knocking;

[0059] For the construction and training of neural network classification models, the existing convolutional neural network models are used, such as Figure 2 As shown in the figure, for the production of the neural network model data set, the footsteps, voices, knocks, glass breaking, and object knocking sounds in a specific security scene are first obtained and saved in the voice database. The main purpose is to use a microphone to receive artificially created sounds of corresponding categories. Each category is saved as an audio file in wav format with two seconds of data as a unit. This is convenient for subsequent processing and ensures that the sound is not distorted, allowing the neural network model to learn more features. Finally, the data set is divided into training and test sets in an 8:2 ratio.

[0060] In the convolutional neural network model, Conv2d+Batch Normalization+Mish is used as the convolution block structure. The introduction of Batch Normalization enables each convolution to obtain a normalization process to avoid the impact of changes in data distribution (Internal Covariate Shift). The Mish function is used as an activation function as shown in formula (1):

[0061] Mish = x*tanh(ln(1+e x )) Formula (1)

[0062] Where x represents the input parameter of the neuron, and Mish represents the function output value, that is, the neuron output after mapping.

[0063] The Mish activation function has an unbounded nature that avoids gradient saturation due to a sharp drop in training speed. The non-monotonic function property helps to maintain small negative values, thereby stabilizing the network gradient flow. The smooth nature gives it good generalization ability and effective optimization ability for results, which can improve the quality of results. Smaller network models can shorten the inference time without losing accuracy to meet the real-time nature of inference.

[0064] In the stage of calculating the loss function, the common cross entropy loss function (Cross Entropy Loss) is not used. For example, the binary classification is shown in formula (2):

[0065]

[0066] Instead, a new loss function Foacl Loss based on cross entropy improvement is used as the loss function for binary classification, as shown in formula (3):

[0067]

[0068] Due to the existence of difficult and easy samples, Foaclloss, as a loss function, effectively mines difficult samples by adding a factor on the basis of the cross entropy loss function, achieving optimal accuracy while ensuring speed.

[0069] Before training the network, set the SGD momentum value, initial weight decay parameters, model iteration times and other hyperparameters, import the preprocessed training and test sets to start training the model, and the model gradually converges on the training and test sets. Once the model loss value on the test set no longer decreases, stop training and obtain the final neural network model (i.e., speech classification model).

[0070] The sound classification module is also used to pre-process the acquired sound, and specifically includes the following submodules:

[0071] Silence removal submodule: used to remove the silence part at the beginning and end of the acquired audio data;

[0072] The sound framing submodule is used to perform framing processing on the audio data after silence removal by means of a moving window function, and to make the overlap between the frames have a specified value; in this embodiment, the specified value is 50%.

[0073] Feature extraction submodule: used to extract features from the framed audio data.

[0074] Data analysis module: used to analyze the number of people and purpose of actions in the current geographical environment based on the results of identification and classification.

[0075] The data analysis module specifically includes the following submodules:

[0076] Semantic analysis submodule: used to perform speech recognition on the acquired audio data, and perform semantic analysis based on the result of speech recognition to obtain the first action purpose;

[0077] Behavior analysis submodule: used to infer and analyze the second action purpose based on the artificially made sound, and to conduct a comprehensive analysis based on the first action purpose and the second action purpose to obtain the action target;

[0078] The number analysis submodule is used to perform a tone analysis on the voices in the man-made sounds to obtain the first number of people; it is also used to perform a tone analysis on the footsteps in the man-made sounds to obtain the second number of people; it is also used to analyze the number range data based on the first number of people and the second number of people.

[0079] A scene recognition method based on speech classification, the above system applies this method, specifically comprising the following steps:

[0080] Sound detection step: The robot detects surrounding sounds and locates the sound source when a sound is detected;

[0081] Mobile control steps: Control the direction of the robot according to the result data of sound source positioning; compared with security personnel and fixed cameras, it can patrol freely without blind spots, and will not feel sleepy in a quiet environment at night, and the accuracy is higher. In case of emergencies, it can find the location in time and go for patrol and investigation.

[0082] Path planning steps: Obtain environmental data through the robot's laser radar, and re-plan the robot's travel direction based on the result data of sound source positioning;

[0083] Lighting control step: When the robot moves in the direction of travel, turn off the robot's lighting equipment. In the path planning step, the use of laser radar is not only convenient for planning the robot's direction of travel in combination with the result data of sound source positioning, but also to prevent the robot from bumping into other objects during the movement, causing damage and noise, thereby disturbing the intruders. The lighting control step is to turn off the robot's lighting equipment when the robot moves in the direction of travel to avoid disturbing the intruders. This makes it easier for the robot to get closer to the intruders to obtain more accurate sound data, thereby ensuring the accuracy of data analysis.

[0084] Sound classification step: while the robot is moving, the acquired audio data is input into a trained neural network classification model for identification and classification, and the identification and classification results include man-made sounds and environmental sounds; the man-made sounds include footsteps, talking, knocking, breaking glass, and objects knocking against each other;

[0085] For the construction and training of neural network classification models, the existing convolutional neural network models are used, such as Figure 2As shown in the figure, for the production of the neural network model data set, the footsteps, voices, knocks, glass breaking, and object knocking sounds in a specific security scene are first obtained and saved in the voice database. The main purpose is to use a microphone to receive artificially created sounds of corresponding categories. Each category is saved as an audio file in wav format with two seconds of data as a unit. This is convenient for subsequent processing and ensures that the sound is not distorted, allowing the neural network model to learn more features. Finally, the data set is divided into training and test sets in an 8:2 ratio.

[0086] In the convolutional neural network model, Conv2d+Batch Normalization+Mish is used as the convolution block structure. The introduction of Batch Normalization enables each convolution to obtain a normalization process to avoid the impact of changes in data distribution (Internal Covariate Shift). The Mish function is used as an activation function as shown in formula (4):

[0087] Mish = x*tanh(ln(1+e x )) Formula (4)

[0088] Where x represents the input parameter of the neuron, and Mish represents the function output value, that is, the neuron output after mapping.

[0089] The Mish activation function has an unbounded nature that avoids gradient saturation due to a sharp drop in training speed. The non-monotonic function property helps to maintain small negative values, thereby stabilizing the network gradient flow. The smooth nature gives it good generalization ability and effective optimization ability for results, which can improve the quality of results. Smaller network models can shorten the inference time without losing accuracy to meet the real-time nature of inference.

[0090] In the stage of calculating the loss function, the common cross entropy loss function (Cross Entropy Loss) is not used. For example, the binary classification is shown in formula (5):

[0091]

[0092] Instead, a new loss function Foacl Loss based on cross entropy improvement is used as the loss function for binary classification, as shown in formula (6):

[0093]

[0094] Due to the existence of difficult and easy samples, Foaclloss, as a loss function, effectively mines difficult samples by adding a factor on the basis of the cross entropy loss function, achieving optimal accuracy while ensuring speed.

[0095] Before training the network, set the SGD momentum value, initial weight decay parameters, model iteration times and other hyperparameters, import the preprocessed training and test sets to start training the model, and the model gradually converges on the training and test sets. Once the model loss value on the test set no longer decreases, stop training and obtain the final neural network model (i.e., speech classification model).

[0096] The sound classification step is also used to pre-process the acquired sound before inputting it into the speech classification model, which specifically includes the following sub-steps:

[0097] Silence removal sub-step: removing the silence parts at the beginning and end of the acquired audio data;

[0098] Sound framing sub-step: Frame the audio data after silence removal by moving the window function, and make the frames overlap by a specified value; in this embodiment, the specified value is 50%.

[0099] Feature extraction sub-step: extract features from the audio data after framing. By preprocessing the acquired sound, it is easy to improve the working efficiency of the neural network model and improve the classification accuracy.

[0100] Data analysis steps: Analyze the number of people and purpose of actions in the current geographical environment based on the results of identification and classification.

[0101] The data analysis step specifically includes the following sub-steps:

[0102] Semantic analysis sub-step: perform speech recognition on the acquired audio data, and perform semantic analysis based on the results of speech recognition to obtain the purpose of the action;

[0103] Behavior analysis sub-step: inferring and analyzing the second action purpose based on the artificially made sound, and performing a comprehensive analysis based on the first action purpose and the second action purpose to obtain the action goal;

[0104] Number analysis sub-step: perform tone analysis on the speaking sound to obtain the first number of people; also perform tone analysis on the footsteps to obtain the second number of people; also perform analysis on the number of people range data based on the first number of people and the second number of people.

[0105] The semantic analysis sub-step and the behavior analysis sub-step make it easy to understand the action target of the intruder, so as to issue an alarm in time or take relevant measures to avoid the problem. The number analysis sub-step makes it easy to analyze the number of intruders, so as to arrange the number of personnel to drive them away.

[0106] A computer-readable storage medium includes a program for scene recognition based on speech classification. When the program for scene recognition based on speech classification is executed by a processor, the steps in a method for scene recognition based on speech classification are implemented.

[0107] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned scene recognition method based on speech classification can be implemented by instructing related hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of various embodiments of the scene recognition method based on speech classification as described above. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0108] Embodiment 2

[0109] The difference between the second embodiment and the first embodiment is that the chassis of the security robot body is also provided with hardware modules such as a camera, a fill light, a GPS positioning module, a light sensor, and an audio and video alarm module. Multiple security robots are remotely connected to the server through the communication module, and wirelessly connected to other security robots to form a network. The server includes the following modules:

[0110] Robot marking module: used to number each security robot; and to record the operating status of each numbered security robot; the security robots that are running are recorded as running, and the security robots that are not running are recorded as standby.

[0111] Patrol arrangement module: used to select security robots according to their numbers and operating status, and set corresponding patrol areas for each security robot.

[0112] Database: used to store environmental map data of security areas;

[0113] Regional environment optimization module: used to obtain the regional environment scanning data collected by the security robot through the lidar module in the corresponding patrol area, and update the environmental map data of the security area according to the regional environment scanning data.

[0114] Position analysis module: used to obtain the positioning information collected by the security robot positioning module, and mark it on the environmental map data together with the security robot's number.

[0115] The software architecture of the robot includes the following modules:

[0116] Image acquisition module: used to collect video image data through a camera toward the sound source position; in this embodiment, the video image data is an RTSP video stream, which is first transmitted to the built-in field algorithm server of the security robot and decoded by the field algorithm server.

[0117] Data analysis module: used to establish reasoning tasks based on the collected video image data, analyze the data calculation amount of the reasoning task, and compare the analysis results of the data calculation amount with its own computing resources. If the computing resources are sufficient, the video image data will be input into the data processing module; if the computing resources are insufficient, a distributed computing task will be issued;

[0118] Task division module: when issuing distributed computing tasks, it is used to divide the reasoning task into several subtasks (including task configuration, reasoning image and other information), and place them in the distributed reasoning queue pool, waiting for other security robots with computing margin to participate in the subtasks; in this embodiment, the communication method between security robots adopts TCP / IP protocol to ensure that the current reasoning task information of the robot can be correctly transmitted and placed in the distributed reasoning queue pool to find other security robots that can be calculated, so that idle security robots can participate in the calculation. The processing results of the subtask after calculation are returned to the security robot that issued the distributed computing task. If there is no idle robot, the computing task enters the queue and waits.

[0119] Data processing module: used to calculate the assigned subtasks to obtain processing results when there is sufficient computing resources, and send them to the corresponding security robots; it is also used to input video image data and / or processing results into the neural network model to perform reasoning tasks and output reasoning results; the reasoning results include security target recognition categories and security target recognition quantities.

[0120] In this implementation, the neural network model specifically adopts a convolutional neural network model. For the collection of the data set, a video camera is used to shoot the video, and then the frames are extracted at intervals, so as to ensure that each image data has no repeated annotation objects, so that the convolutional neural network model can learn more features. For the annotation of the data set, the annotation categories are pre-defined. For example, in the river area, all possible targets can be divided into ordinary people category A, blurred face personnel category B, open flame category C, smoke category D, etc. According to the above classification, the image is annotated with a border, and the annotation result is the information and position information of each category on the image, that is, the two-dimensional coordinates of the upper left corner vertex of the rectangle (x0, y0), and the two-dimensional coordinates of the lower right corner vertex (x1, y1). For the training and testing of the convolutional neural network model, the data set is divided into a training set, a test set, and a validation set in a ratio of 7:2:1. Then the training is continuously iterated to finally obtain a trained convolutional neural network model.

[0121] In this embodiment, during the training phase of the convolutional neural network model, the conventional LeakyRelu activation function is modified to a hard-swish function, and its formula is as follows:

[0122]

[0123] This method improves the nonlinear ability of the activation function and has a stronger ability to extract features in deeper network layers.

[0124] Moreover, during the model training process, noise images such as multi-size cropping, different gain multiples, and mosaic occlusion can be added to the data set to make the convolutional neural network model more generalizable. This will effectively reduce the possibility that the confidence of the target category will oscillate between high and low ranges. As for the mosaic data enhancement, the method is to randomly mosaic occlude the photos in the data set. This method significantly increases the number of data sets while improving the generalization ability of more data and effectively preventing model overfitting. The implementation method of hybrid enhancement used in target detection is: the two images are fused with RGB values ​​at a certain ratio, and the model is required to predict all the targets in the original two images.

[0125] Alarm processing module: used to perform alarm analysis based on the inference results and analyze and determine whether to generate alarm information.

[0126] The alarm processing module includes the following submodules:

[0127] Alarm generation submodule: used to determine whether the security target identification category is an alarm category, and if so, to generate alarm information; also used to analyze and determine the current alarm level based on the number of security target identifications and the preset alarm level range;

[0128] Sound and light warning submodule: used to issue corresponding light and sound alarms according to the alarm information and the current alarm level. For example, if the security target detection category is found to be one of the ones that require alarms, and the level is medium; first, the robot's external speaker shouts loudly to scare away suspicious persons, and the robot's chassis light strip flashes red and green.

[0129] Light compensation module: used to obtain ambient light intensity through the light sensor, and start the fill light when the ambient light intensity is lower than the light intensity threshold, so that the background monitoring personnel can view the alarm scene in real time.

[0130] Video and photo module: used to take videos and photos of security targets through cameras and upload them to the server for storage, ensuring the use of later traceability and evidence collection.

[0131] The server also includes an alarm response module, which is used to search for other surrounding security robots from environmental map data based on the positioning information of the security robot when the security robot generates an alarm message, and to make other security robots move toward the position of the security robot according to the positioning information.

[0132] The method further comprises the following steps:

[0133] Robot marking steps: number each security robot; and record the operating status of each numbered security robot; the security robots that are running are recorded as running, and the security robots that are not running are recorded as standby.

[0134] Patrol arrangement steps: select security robots according to their numbers and operating status, and set corresponding patrol areas for each security robot.

[0135] Storage step: used to store environmental map data of the security area;

[0136] Regional environment optimization steps: obtain the regional environment scanning data collected by the security robot through the laser radar module in the corresponding patrol area, and update the environmental map data of the security area according to the regional environment scanning data.

[0137] Position analysis steps: obtain the positioning information collected by the security robot positioning module, and mark it on the environmental map data together with the security robot's number.

[0138] The software architecture of the robot includes the following modules:

[0139] Image acquisition step: collect video image data by pointing the camera toward the sound source; in this embodiment, the video image data is an RTSP video stream, which is first transmitted to the built-in on-site algorithm server of the security robot and decoded by the on-site algorithm server.

[0140] Data analysis step: establish reasoning tasks based on the collected video image data, analyze the data calculation amount of the reasoning tasks, and compare the analysis results of the data calculation amount with its own computing resources. If the computing resources are sufficient, the video image data will be input into the data processing module; if the computing resources are insufficient, a distributed computing task will be issued;

[0141] Task division step: When issuing a distributed computing task, the reasoning task is divided into several subtasks (including task configuration, reasoning image and other information), and placed in the distributed reasoning queue pool, waiting for other security robots with computing margin to participate in the subtask; in this embodiment, the communication method between security robots adopts TCP / IP protocol to ensure that the current reasoning task information of the robot can be correctly transmitted and placed in the distributed reasoning queue pool to find other security robots that can be calculated, so that idle security robots can participate in the calculation. The processing results of the calculated subtask are returned to the security robot that issued the distributed computing task. If there is no idle robot, the computing task enters the queue and waits.

[0142] Data processing steps: When computing resources are sufficient, the assigned subtasks are calculated to obtain processing results, and sent to the corresponding security robots; it is also used to input video image data and / or processing results into the neural network model to perform reasoning tasks and output reasoning results; the reasoning results include security target recognition categories and security target recognition quantities.

[0143] Alarm processing steps: Perform alarm analysis based on the inference results to determine whether to generate alarm information.

[0144] The alarm processing module includes the following submodules:

[0145] Alarm generation sub-step: determining whether the security target identification category is an alarm category, and if so, generating alarm information; and also used to analyze and determine the current alarm level according to the number of security target identifications and the preset alarm level interval;

[0146] Sound and light warning sub-step: according to the alarm information and the current alarm level, the corresponding level of light alarm and sound alarm is carried out. For example, if the security target detection category is found to be one of the ones that need to be alarmed, and the level is medium; first, the robot's external speaker shouts loudly to scare away the suspicious person, and the robot's chassis light strip flashes red and green.

[0147] Light compensation steps: obtain the ambient light intensity through the light sensor, and start the fill light when the ambient light intensity is lower than the light intensity threshold, so that the background monitoring personnel can view the alarm scene in real time.

[0148] Video and photo taking steps: Use the camera to take videos and photos of the security target and upload them to the server for storage. This ensures that they can be used for later tracing and evidence collection.

[0149] Alarm response step: when the security robot generates an alarm message, other surrounding security robots are searched from the environmental map data according to the positioning information of the security robot, and other security robots are moved toward the position of the security robot according to the positioning information.

[0150] The above is only an embodiment of the present invention. The common sense such as the known specific structure and characteristics in the scheme is described too much here. The ordinary technicians in the relevant field know all the common technical knowledge in the technical field of the invention before the application date or priority date, can know all the existing technologies in the field, and have the ability to apply the conventional experimental means before that date. The ordinary technicians in the relevant field can improve and implement this scheme in combination with their own abilities under the enlightenment given by this application. Some typical known structures or known methods should not become obstacles for ordinary technicians in the relevant field to implement this application. It should be pointed out that for those skilled in the art, without departing from the structure of the present invention, several deformations and improvements can be made, which should also be regarded as the protection scope of the present invention, which will not affect the effect of the implementation of the present invention and the practicality of the patent. The protection scope required by this application shall be based on the content of its claims, and the specific implementation methods and other records in the specification can be used to interpret the content of the claims.

Claims

1. A scene recognition system based on speech classification, characterized by: The robot includes a robot and a server. Multiple robots are remotely connected to the server through a communication module, and are connected to other robots through wireless communication to form a network. The robot includes the following modules: Sound detection module: used to detect surrounding sounds and locate the sound source when sound is detected; Mobile control module: used to control the robot's moving direction according to the result data of sound source positioning; Sound classification module: used to input the acquired audio data into the trained neural network classification model for recognition and classification during the robot's continuous movement. The recognition and classification results include artificial sounds and environmental sounds. Image acquisition module: used to collect video image data through a camera toward the sound source; Data analysis module: used to analyze the number of people and the purpose of action in the current geographical environment according to the results of recognition and classification; establish reasoning tasks according to the collected video image data, analyze the data calculation amount of the reasoning task, and compare and analyze the analysis results of the data calculation amount with its own computing resources. If the computing resources are sufficient, the video image data will be input into the data processing module; If computing resources are insufficient, distributed computing tasks are issued; Task division module: used to divide the reasoning task into several subtasks when issuing distributed computing tasks, including task configuration and reasoning images, and place them in the distributed reasoning queue pool, waiting for other robots with computing margin to participate in the subtasks; After the calculation, the processing result of the subtask is returned to the robot that issued the distributed computing task. If there is no idle robot, the computing task will enter the queue and wait; Data processing module: used to calculate the assigned subtasks to obtain processing results when computing resources are sufficient, and send them to the corresponding robots; also used to input video image data and / or processing results into the neural network model to perform reasoning tasks, and output reasoning results; the reasoning results include security target recognition categories and security target recognition quantities; Alarm processing module: used to analyze alarms based on the inference results and determine whether to generate alarm information; Light compensation module: used to obtain the ambient light intensity through the light sensor, and start the fill light when the ambient light intensity is lower than the light intensity threshold; Video and photo module: used to take videos and photos of security targets through the camera and upload them to the server for storage; The server includes the following modules: Robot marking module: used to number each robot and record the running status of each numbered robot, the running status includes running and standby; Patrol Arrangement Module: used to select robots according to their numbers and operating status, and set corresponding patrol areas for each robot; Database: used to store environmental map data of security areas; Regional environment optimization module: used to obtain the regional environment scanning data collected by the robot through the laser radar module in the corresponding patrol area, and update the environmental map data of the security area according to the regional environment scanning data; Position analysis module: used to obtain the positioning information collected by the robot positioning module, and mark it on the environmental map data together with the robot number; The alarm response module is used to search for other robots around the robot from the environmental map data according to the positioning information of the robot when the robot generates an alarm message, and to make the other robots move toward the position of the robot according to the positioning information.

2. The scene recognition system based on speech classification according to claim 1, characterized in that: Also includes the following modules: Path planning module: used to obtain environmental data through the robot's laser radar, and re-plan the robot's travel direction based on the result data of sound source positioning; Light control module: used to turn off the robot's lighting equipment when the robot moves according to the direction of travel.

3. The scene recognition system based on speech classification according to claim 1, characterized in that: The sound classification module is also used to pre-process the acquired sound, and specifically includes the following submodules: Silence removal submodule: used to remove the silence part at the beginning and end of the acquired audio data; Sound framing submodule: used to perform framing processing on the audio data after silence removal through a moving window function, and to make the frames overlap by a specified value; Feature extraction submodule: used to extract features from the framed audio data.

4. The scene recognition system based on speech classification according to claim 1, characterized in that: The data analysis module specifically includes the following submodules: Semantic analysis submodule: used to perform speech recognition on the acquired audio data, and perform semantic analysis based on the result of speech recognition to obtain the first action purpose; Behavior analysis submodule: used to infer and analyze the second action purpose based on the artificially made sound, and to conduct a comprehensive analysis based on the first action purpose and the second action purpose to obtain the action target; The number analysis submodule is used to perform a tone analysis on the speech sound in the artificially produced sound to obtain the first number of people; It is also used to perform tone analysis on footsteps in artificially created sounds to obtain the second number of people; it is also used to analyze the number range data based on the first number of people and the second number of people.

5. A scene recognition method based on speech classification, characterized in that: The following steps are involved: Sound detection step: The robot detects surrounding sounds and locates the sound source when a sound is detected; Mobile control step: Control the robot's moving direction according to the result data of sound source positioning; Sound classification step: while the robot is moving, the acquired audio data is input into a trained neural network classification model for recognition and classification, wherein the recognition and classification results include artificially created sounds and environmental sounds; Image acquisition step: collect video image data by pointing the camera toward the sound source; Data analysis steps: Analyze the number of people and the purpose of action in the current geographical environment based on the results of recognition and classification; establish reasoning tasks based on the collected video image data, analyze the data calculation amount of the reasoning tasks, and compare and analyze the analysis results of the data calculation amount with its own computing resources. If the computing resources are sufficient, the video image data will be input into the data processing module; If computing resources are insufficient, distributed computing tasks are issued; Task division step: When issuing distributed computing tasks, the reasoning task is divided into several subtasks, including task configuration and reasoning images, and placed in the distributed reasoning queue pool, waiting for other robots with computing margin to participate in the subtasks; After the calculation, the processing result of the subtask is returned to the robot that issued the distributed computing task. If there is no idle robot, the computing task will enter the queue and wait; Data processing step: when computing resources are sufficient, the assigned subtasks are calculated to obtain processing results, and sent to the corresponding robots; it is also used to input the video image data and / or processing results into the neural network model to perform reasoning tasks, and output reasoning results; the reasoning results include security target recognition categories and security target recognition quantities; Alarm processing steps: Perform alarm analysis based on the inference results to determine whether to generate alarm information; Light compensation steps: obtain the ambient light intensity through the light sensor, and start the fill light when the ambient light intensity is lower than the light intensity threshold; Video and photo taking steps: Use the camera to take videos and photos of the security target and upload them to the server; Robot marking step: the server numbers each robot and records the running status of each numbered robot, wherein the running status includes running and standby; Patrol arrangement steps: The server selects robots according to their numbers and operating status, and sets corresponding patrol areas for each robot; Data storage step: The server stores the environmental map data of the security area; Regional environment optimization step: The server obtains the regional environment scanning data collected by the robot through the laser radar module in the corresponding patrol area, and updates the environmental map data of the security area according to the regional environment scanning data; Position analysis step: The server obtains the positioning information collected by the robot positioning module and marks it on the environmental map data together with the robot number; Alarm response step: When the robot generates an alarm message, the server searches for other robots around it from the environmental map data based on the positioning information of the robot, and makes the other robots move toward the position of the robot based on the positioning information.

6. The scene recognition method based on speech classification according to claim 5, characterized in that: The following steps are also included: Path planning steps: Obtain environmental data through the robot's laser radar, and re-plan the robot's travel direction based on the result data of sound source positioning; Light control steps: When the robot moves according to the direction of travel, turn off the robot's lighting equipment.

7. The scene recognition method based on speech classification according to claim 5, characterized in that: The sound classification step is also used to pre-process the acquired sound, and specifically includes the following sub-steps: Silence removal sub-step: removing the silence parts at the beginning and end of the acquired audio data; Sound framing sub-step: framing the audio data after silence removal by moving the window function, and making the overlap between the frames have a specified value; Feature extraction sub-step: extract features from the audio data that has completed framing.

8. The scene recognition method based on speech classification according to claim 5, characterized in that: The data analysis step specifically includes the following sub-steps: Semantic analysis sub-step: perform speech recognition on the acquired audio data, and perform semantic analysis based on the results of speech recognition to obtain the purpose of the action; Behavior analysis sub-step: inferring and analyzing the second action purpose based on the artificially made sound, and performing a comprehensive analysis based on the first action purpose and the second action purpose to obtain the action goal; Number analysis sub-step: performing tone analysis on the speaking voice to obtain the number of the first person; It is also used to perform tone analysis on footsteps to obtain the second number of people; it is also used to analyze the number range data based on the first number of people and the second number of people.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a program for scene recognition based on speech classification. When the program for scene recognition based on speech classification is executed by a processor, the steps of the scene recognition method based on speech classification as claimed in any one of claims 5 to 8 are implemented.

Images