Human body recognition method and system based on multi-modal sensor

Abstract

The invention relates to a human body recognition method and system based on a multi-modal sensor. The method comprises the following steps: detecting a start instruction in real time to control each sensor to perform data acquisition operation, acquiring data of each sensor for analysis, and entering a system standby interaction state if entering a system wake-up state; performing human body recognition according to the data of each sensor, receiving input voice data after the human body recognition succeeds, and executing a corresponding voice instruction according to the voice data; and if the input voice data is not received within the threshold time period after successful human body recognition, entering a standby state, and detecting a starting instruction in real time. When human body recognition is carried out, the multi-mode sensor can effectively cope with the problems of light change, shielding and human face postures, recognition failures caused by environmental factors are reduced, and human body recognition can be efficiently, stably and accurately achieved; the multi-mode sensors are matched with one another, the data processing flow is optimized, and system resource consumption is reduced.

Description

Technical Field

[0001] The present invention relates to the field of intelligent recognition technology, and in particular to a human body recognition method and system based on a multimodal sensor. Background Art

[0002] In the current commercial robotics and intelligent recognition fields, single-face recognition technology has been widely used, and its technical principles and basic processes have been disclosed in numerous academic research studies, commercial product documentation, and open source projects. The core process of single-face recognition technology is usually to first use a camera to capture a facial image, then preprocess the image (such as grayscale and normalization), and then obtain the facial feature vector through a feature extraction algorithm (such as traditional algorithms such as PCA and LDA or deep learning-based network architectures such as ResNet and VGG). Finally, it is compared with a pre-stored facial feature database to determine the identity or determine whether the recognition is successful. In actual application scenarios, such as airport security and access control systems, single-face recognition technology can meet basic identity authentication needs to a certain extent, but its limitations are gradually becoming apparent in complex public environments.

[0003] In terms of light, single face recognition has strict requirements on lighting conditions. When in direct strong light (such as strong sunlight outdoors) or low light environment (such as a dimly lit corridor at night), the brightness and contrast of the facial image will change drastically, making feature extraction difficult. For example, in strong light, the face may have overexposed areas, resulting in the loss of facial details; in low light, image noise increases, and facial features become blurred. Through actual testing, the same set of facial images were recognized under different light intensities. The recognition accuracy in strong light environment dropped by about 30%-40% compared to normal light conditions, and in low light environment it dropped by about 40%-50%.

[0004] Regarding facial posture and occlusion, when a person's face is not directly in front of the camera (e.g., facing in profile, looking down, or looking up at a large angle), the facial image captured by the camera will be distorted, making it difficult for traditional feature extraction algorithms to accurately extract effective feature information. According to statistics, when the face deflection angle exceeds 30 degrees, the recognition accuracy rate will drop below 50%. At the same time, in real-world scenarios, people often wear items such as masks, glasses, and hats. These obstructions will partially or completely block key areas of the face, seriously affecting the accuracy of face recognition.

[0005] From a system performance perspective, as facial recognition tasks continue to run, their CPU utilization continues to climb. This is because facial recognition algorithms require complex computation and data storage operations when processing large amounts of facial image data. For example, in a surveillance scenario requiring simultaneous processing of 10 camera streams, the facial recognition system's CPU utilization can reach as high as 70%-80% after a period of operation, causing system lag and significantly increasing recognition latency, severely impacting the system's real-time performance and responsiveness.

[0006] Therefore, traditional intelligent human recognition methods are often affected by light, facial posture and occlusion in complex public scenes, and have long running times, high system CPU usage, resulting in recognition delays and low recognition accuracy. Summary of the Invention

[0007] Based on this, in order to solve the above technical problems, a human body recognition method and system based on multimodal sensors are provided, which can realize human body recognition efficiently, stably and accurately, and reduce system resource consumption.

[0008] A human body recognition method based on a multimodal sensor, the method comprising:

[0009] Detecting the start instruction in real time, controlling each sensor to perform data acquisition operations according to the start instruction, and acquiring data from each sensor;

[0010] Analyze the sensor data and determine whether to enter the system awake state according to the analysis result. If the system awake state is entered, enter the system standby interactive state;

[0011] In the standby interactive state of the system, human body recognition is performed according to the data of each sensor, and after successful human body recognition, input voice data is received and corresponding voice instructions are executed according to the voice data;

[0012] If no voice data is received within a threshold period after human body recognition is successful, the system enters a standby state and detects the start command in real time.

[0013] In one embodiment, detecting a start instruction in real time and controlling each sensor to perform data acquisition operations according to the start instruction include:

[0014] monitoring environmental sound data in real time, analyzing the environmental sound data in real time, and obtaining analysis results;

[0015] Determining whether a wake-up word appears in the ambient sound data according to the analysis result;

[0016] If the wake-up word appears, the touch signal is continuously detected, and the detected touch signal is used as a start instruction for each sensor to perform data collection operations;

[0017] If the wake-up word does not appear, continue to monitor the ambient sound data.

[0018] In one embodiment, detecting a start instruction in real time and controlling each sensor to perform data acquisition operations according to the start instruction include:

[0019] Collect gesture image information in real time through a depth camera or infrared camera;

[0020] performing gesture recognition on the gesture image information based on image recognition technology to obtain a gesture recognition result;

[0021] Control each sensor to perform data collection operations according to the gesture recognition result.

[0022] In one embodiment, each of the sensors includes a camera, an ultrasonic sensor, a lidar, and an RGBD depth camera; obtaining data from each sensor includes:

[0023] Acquiring image data collected by the camera;

[0024] Acquiring a reflected wave received after the ultrasonic sensor transmits the ultrasonic wave;

[0025] Acquiring point cloud data collected by scanning the surrounding environment through the laser radar; and

[0026] Acquire depth image and color image information acquired by the RGBD depth camera.

[0027] In one embodiment, analyzing the sensor data and determining whether to enter the system awake state based on the analysis results includes:

[0028] Analyzing the image data, reflected wave, point cloud data, depth image, and color image information respectively to obtain analysis results;

[0029] If the analysis result is that the image data contains a human face and the confidence level is higher than a set threshold; or

[0030] The reflected wave meets the trigger condition; or

[0031] The point cloud data includes human body feature point cloud data; or

[0032] If the depth image and color image information contain a human target, the system enters a wake-up state.

[0033] In one embodiment, when the system is in a standby interactive state, human body recognition is performed based on the sensor data, and input voice data is received after successful human body recognition, including:

[0034] In the standby interactive state of the system, if the image data contains a human face and the confidence level is higher than a set threshold; or

[0035] The reflected wave meets the trigger condition; or

[0036] The point cloud data includes human body feature point cloud data; or

[0037] If the depth image and color image information contain a human target, the human body recognition is successful and the input voice data is received.

[0038] In one embodiment, the method further comprises:

[0039] If the system is in a wake-up state or a standby interactive state, the sensor interface corresponding to each of the sensors is activated;

[0040] If the system is not in the awake state or the standby interactive state, the sensor interface corresponding to each sensor is closed.

[0041] In one embodiment, the method further comprises:

[0042] Obtaining a sensor parameter adjustment instruction and displaying a parameter setting interface according to the sensor parameter adjustment instruction;

[0043] Adjustment parameters are collected through the parameter setting interface, and parameters of each of the sensors are adjusted according to the adjustment parameters.

[0044] A human body recognition system based on a multimodal sensor, the system comprising:

[0045] The sensor data acquisition module is used to detect the start instruction in real time, control each sensor to perform data acquisition operations according to the start instruction, and obtain data from each sensor;

[0046] A system wake-up module is used to analyze the data of each sensor and determine whether to enter the system wake-up state according to the analysis result. If the system enters the system wake-up state, the system enters the standby interactive state;

[0047] An instruction execution module is used to perform human body recognition based on the data from each sensor when the system is in a standby interactive state, and receive input voice data after successful human body recognition, and execute corresponding voice instructions based on the voice data;

[0048] The command detection module is used to enter a standby state and detect a start command in real time if no input voice data is received within a threshold time period after human body recognition is successful.

[0049] In one embodiment, the sensor data acquisition module is also used to monitor the ambient sound data in real time, perform real-time analysis on the ambient sound data, and obtain analysis results; determine whether a wake-up word appears in the ambient sound data based on the analysis results; if the wake-up word appears, continuously detect touch signals, and use the detected touch signals as a start-up instruction for each sensor to perform data acquisition operations; if the wake-up word does not appear, continue to monitor the ambient sound data.

[0050] The above-mentioned human body recognition method and system based on multimodal sensors obtains sensor data by using various multimodal sensors. When performing human body recognition, the multimodal sensor combination can effectively deal with light changes, occlusions and facial posture problems, reduce recognition failures caused by environmental factors, and can realize human body recognition efficiently, stably and accurately; multimodal sensors can cooperate with each other to optimize the data processing process and reduce system resource consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 FIG1 is a diagram showing an application environment of a human body recognition method based on a multimodal sensor in one embodiment;

[0052] Figure 2 1 is a flow chart of a method for human body recognition based on a multimodal sensor in one embodiment;

[0053] Figure 3 is a schematic diagram of the triggering mechanism of different sensors in one embodiment;

[0054] Figure 4 A schematic diagram of an identification range in one embodiment;

[0055] Figure 5 is an application flow chart of a human body recognition method based on a multimodal sensor in one embodiment;

[0056] Figure 6 is a structural block diagram of a human body recognition system based on a multimodal sensor in one embodiment;

[0057] Figure 7 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0059] The human body recognition method based on multimodal sensor provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Figure 1 As shown, the application environment includes a computer device 110. The computer device 110 can detect startup instructions in real time, control various sensors to perform data collection operations based on the startup instructions, and obtain data from various sensors. The computer device 110 can analyze the data from various sensors and determine whether to enter the system wake-up state based on the analysis results. If the system wake-up state is entered, the computer device 110 enters the system standby interactive state. In the system standby interactive state, the computer device 110 can perform human body recognition based on the data from various sensors, and receive input voice data after successful human body recognition, and execute corresponding voice instructions based on the voice data. If no input voice data is received within a threshold time period after successful human body recognition, the computer device 110 can enter the standby state and detect startup instructions in real time. The computer device 110 can be, but is not limited to, various personal computers, laptops, smart phones, robots, unmanned aerial vehicles, tablet computers, and other devices.

[0060] In one embodiment, Figure 2 As shown, a human body recognition method based on a multimodal sensor is provided, comprising the following steps:

[0061] Step 202 : detecting the start instruction in real time, controlling each sensor to perform data acquisition operations according to the start instruction, and acquiring data from each sensor.

[0062] The computer device can detect in real time whether there is a start instruction, and the start instruction can be used to start each sensor to collect data.

[0063] In one embodiment, a human body recognition method based on a multimodal sensor is provided, which may also include a process of controlling each sensor to start working. The specific process includes: real-time monitoring of ambient sound data, real-time analysis of the ambient sound data, and obtaining analysis results; judging whether a wake-up word appears in the ambient sound data based on the analysis results; if a wake-up word appears, continuously detecting touch signals, and using the detected touch signals as start-up instructions for each sensor to perform data collection operations; if no wake-up word appears, continuing to monitor the ambient sound data.

[0064] The computer device can be equipped with a voice recognition module. After the entire system is started, the voice recognition module continuously monitors the ambient sound and performs real-time analysis of the input audio signal to determine whether a preset wake-up word is present. If the wake-up word is detected, further touch detection is performed; if the wake-up word is not detected, the monitoring continues.

[0065] The computer device may also be provided with a touch sensing module that continuously monitors whether the screen is being touched. If a touch operation is detected, each multimodal sensor simultaneously starts data collection; if no touch operation is detected, the monitoring continues.

[0066] In addition to voice wake-up and screen touch wake-up, gesture recognition wake-up can also be added. Specifically, in one embodiment, the wake-up operation can also include: real-time acquisition of gesture image information through a depth camera or infrared camera; gesture recognition based on the gesture image information using image recognition technology to obtain gesture recognition results; and control of various sensors to perform data acquisition operations based on the gesture recognition results.

[0067] The system uses image recognition technology to identify specific user gestures (such as waving or clenching a fist) by installing a depth camera or infrared camera on the computer device. A gesture recognition module is added to the computer program to analyze and identify gesture images using deep learning image classification or object detection algorithms. When a preset wake-up gesture is recognized, a wake-up signal is sent to the system to start the service process of the computer device.

[0068] In one embodiment, each sensor includes a camera, an ultrasonic sensor, a lidar, and an RGBD depth camera; the provided human body recognition method based on a multimodal sensor may also include a process of obtaining data from each sensor through each sensor, and the specific process includes: obtaining image data collected by the camera; obtaining reflected waves received after ultrasonic waves are emitted by the ultrasonic sensor; obtaining point cloud data collected by scanning the surrounding environment through the lidar; and obtaining depth image and color image information obtained through the RGBD depth camera.

[0069] After voice or touch wake-up is triggered, the camera, ultrasonic sensor, lidar, and RGBD depth camera start collecting data simultaneously. The triggering mechanism and corresponding status of each sensor are as follows: Figure 3 As shown in the figure, the camera collects image data at a set frame rate (e.g., 30 frames per second) and transmits the data to the subsequent processing unit; the ultrasonic sensor emits ultrasonic waves at a specific frequency (e.g., 40kHz) and prepares to receive reflected waves; the lidar scans the surrounding environment at a specified frequency (e.g., 10Hz) and collects point cloud data; and the RGBD depth camera obtains depth and color image information.

[0070] Specifically, in this embodiment, when collecting sensor data, the camera adjusts the focal length to a preset value (such as 50mm), starts to collect image data at a frame rate of 30 frames per second, and uses the built-in image signal processor to reduce noise and enhance the image, and then transmits the processed image data to the face detection module; the ultrasonic sensor emits ultrasonic pulses at a frequency of 40kHz, and starts the timer at the same time, waiting for the reception of the reflected wave; the lidar emits a laser beam to the surrounding 360-degree space at a frequency of 10Hz, and obtains point cloud data through scanning with a rotating mirror, and transmits the point cloud data to the subsequent analysis module after preliminary filtering; the RGBD depth camera synchronously obtains depth image and color image information, uses a deep convolutional neural network to analyze the image, extracts feature information of potential human targets, and transmits the data to the system.

[0071] Compared to traditional single-face recognition, the multimodal sensor combination can effectively cope with lighting changes, occlusion, and facial posture issues. In complex lighting environment tests, the accuracy of single-face recognition can be as low as 30%-50%, while the recognition method in this embodiment can maintain a stable accuracy of over 85%. In occlusion scenarios, the recognition success rate of traditional methods is less than 40%, while this embodiment can still maintain a high level of around 70%, greatly reducing recognition failures caused by environmental factors, and can stably sense the approach of the human body, improving the timeliness and reliability of service activation.

[0072] In another embodiment, millimeter-wave radar can be considered as a replacement for lidar. Millimeter-wave radar also has high accuracy in detecting human motion and distance. It operates by transmitting electromagnetic waves in the millimeter-wave frequency band and receiving reflected waves to obtain information about the target object. In this embodiment, the millimeter-wave radar's driver and data processing algorithm need to be adjusted accordingly. The driver must control the millimeter-wave radar to transmit electromagnetic waves at an appropriate frequency and power, and receive and process the reflected signals. The data processing algorithm, based on the characteristics of millimeter-wave radar, analyzes the frequency, phase, and other information of the reflected signals to extract the characteristics and distance data of the human target. When a human target that meets the requirements is detected, a trigger signal is sent to the system.

[0073] Step 204 : Analyze the data of each sensor and determine whether to enter the system awake state according to the analysis result. If the system enters the system awake state, the system enters the system standby interactive state.

[0074] After obtaining data from various sensors, the computer device can determine whether to enter a system wake-up state based on the obtained data from various sensors, thereby deciding whether to perform subsequent processing.

[0075] In one embodiment, a human body recognition method based on a multimodal sensor is provided, which may also include a process of analyzing sensor data. The specific process includes: analyzing the image data, reflected wave, point cloud data, depth image and color image information respectively to obtain analysis results; if the analysis result is that the image data contains a face and the confidence level is higher than a set threshold; or the reflected wave meets the trigger condition; or the point cloud data contains human feature point cloud data; or the depth image and color image information contain human targets, then the system enters a wake-up state.

[0076] The computer device can make a comprehensive judgment on the data collected by each sensor. Specifically, if the camera detects a human face with a confidence level higher than a set threshold (such as 80%), or the ultrasonic sensor receives a reflected wave that meets the trigger conditions, or the lidar identifies point cloud data that meets human characteristics, or the RGBD depth camera determines the presence of a human target, as long as one of these conditions is met, the wake-up is determined to be successful and enters the standby interaction preparation state. If all sensors are not triggered, the wake-up monitoring state will continue.

[0077] Specifically, in this embodiment, the computer device can receive data from various sensors. The face detection module in the computer device analyzes the camera image and determines that the wake-up is successful if a face is detected with a confidence level greater than 80%. The ultrasonic sensor determines whether an object is approaching based on the time difference and intensity change of the reflected wave. If a reflected wave with an intensity change exceeding a threshold is detected within a range of 0.5-2.5m, the wake-up is also determined to be successful. The lidar analyzes the point cloud data and determines that the wake-up is successful if a point cloud cluster that matches the human outline and motion characteristics is identified. The RGBD depth camera also determines that the wake-up is successful if, based on the image analysis results, it determines that a human target is present and the distance is within the effective range. As long as at least one sensor meets the wake-up condition, the system enters the standby interactive state. If none of the sensors trigger the wake-up condition, the wake-up monitoring state remains.

[0078] Multimodal recognition optimizes data processing by intelligently assigning sensor tasks. Experimental comparisons show that after 24 hours of continuous operation, the CPU utilization of traditional facial recognition systems can reach as high as 70%-80%, resulting in a recognition delay of 1-2 seconds. This embodiment reduces CPU utilization to 40%-50%, reducing recognition delay to less than 0.5 seconds. This significantly improves the system's overall performance and response speed, ensuring that the robot can provide users with fast and smooth service even under high load.

[0079] Step 206 , in the system standby interactive state, human body recognition is performed based on the data from various sensors, and after successful human body recognition, input voice data is received and corresponding voice commands are executed based on the voice data.

[0080] In the standby interaction preparation state, each sensor continuously collects data and transmits it to the system. The system receives image data from the camera, reflected wave data from the ultrasonic sensor, point cloud data from the lidar, and image and depth data from the RGBD depth camera.

[0081] In one embodiment, a human body recognition method based on a multimodal sensor is provided, which may also include a process for performing human body recognition. The specific process includes: in the system standby interaction state, if the image data contains a human face and the confidence level is higher than a set threshold; or the reflected wave meets the trigger condition; or the point cloud data contains human feature point cloud data; or the depth image and color image information contain human targets, then the human body recognition is successful and the input voice data is received.

[0082] The computer device can analyze the data from each sensor. If any sensor data indicates a trigger state (such as a camera detecting a face, an ultrasonic sensor detecting an approaching object, a lidar identifying a human point cloud, or an RGBD depth camera identifying a potential human target), the voice interaction function is activated. If all sensor data indicates an untriggered state, the system continues to wait for data collection and analysis.

[0083] Among them, in the human body recognition judgment logic, in addition to the "or" logic (as long as one sensor is triggered, it will take effect), weighted judgment logic can also be used. Different weights are assigned according to the reliability and accuracy of different sensors. For example, the weight of camera face detection is 0.4, the weight of RGBD depth camera is 0.3, the weight of ultrasonic sensor is 0.2, and the weight of lidar is 0.1. When the sum of the weights corresponding to the trigger signals of each sensor exceeds the set threshold (such as 0.5), human body recognition is determined to be effective. In the computer program, a weight calculation and judgment module can be added to receive the trigger signals of each sensor, and calculate and judge according to the preset weights to decide whether to start the subsequent process.

[0084] Step 208: If no voice data is received within a threshold period of time after human body recognition is successful, the system enters a standby state and detects a start instruction in real time.

[0085] After human recognition is determined to be successful, the voice interaction function can be activated. The speech synthesis module in the computer device prepares to respond, the sound pickup module begins to receive user voice input, and a 30-second timer is started. If user voice input is detected within 30 seconds, the corresponding voice processing and task execution steps are executed. If there is no user interaction within 30 seconds, the voice stop module is called to stop sound pickup and broadcasting, shut down the voice interaction function, return to standby mode, and continue human recognition monitoring.

[0086] Among them, after the computer device enters the voice interaction preparation state, it can start the voice interaction process and communicate with the user by voice. During the communication process, it can continuously monitor the timer for 30 seconds. If the user has a new voice command or interactive behavior within 30 seconds, it will continue to process and respond; if there is no interaction within 30 seconds, the voice interaction function will be stopped, unnecessary sensor modules will be turned off or their operating frequency will be reduced to save energy, and the computer device will enter the standby state. At the same time, the computer device can regularly (such as every 10 seconds) check the sensor status and wake-up conditions. Once a wake-up word, touch operation or human proximity signal is detected, the normal working parameters of each sensor and the voice interaction function will be immediately restored, and a new round of human recognition and interaction process will be restarted.

[0087] In one embodiment, a human body recognition method based on a multimodal sensor is provided, which may also include a process of providing an interface for each sensor. The specific process includes: if the system is in a wake-up state or a system standby interactive state, activating the sensor interface corresponding to each sensor; if the system is not in a wake-up state or a system standby interactive state, closing the sensor interface corresponding to each sensor.

[0088] Computer devices can monitor their operating status in real time to determine whether they are currently in standby mode, voice interaction, customer service, or navigation tasks. If they are in standby mode or voice interaction mode, they activate the interfaces of multimodal human recognition sensors to enable them to operate normally and transmit data. If they are in customer service or navigation tasks, they disable the interfaces of these sensors and stop receiving sensor data, thus avoiding resource waste and data interference.

[0089] In one embodiment, a human body recognition method based on a multimodal sensor is provided, which may also include a process of adjusting sensor parameters. The specific process includes: obtaining a sensor parameter adjustment instruction, and displaying a parameter setting interface according to the sensor parameter adjustment instruction; collecting adjustment parameters through the parameter setting interface, and adjusting the parameters of each sensor according to the adjustment parameters.

[0090] The user can input the desired recognition distance and angle parameters through the setting interface of the computer device. Figure 4As shown, users can set the recognition distance to 1.5m-2m and adjust the angle to 60 degrees (30 degrees left and right). After receiving the setting instructions, the computer device will adjust the working parameters of each sensor accordingly. Specifically, the camera's focal length and field of view can be adjusted so that it can clearly capture facial and human features at a distance of 1.5m-2m, and the horizontal viewing angle is limited to 60 degrees. The ultrasonic sensor reduces the transmission power and improves the receiving sensitivity, accurately adjusting the effective detection distance to within 1.5m-2m, and adjusting the transmission and receiving directions according to the angle setting. The lidar reconfigures the scanning parameters, narrowing the scanning range to a 60-degree fan-shaped area and improving the point cloud resolution within this area to ensure accurate recognition of the human body at a distance of 1.5m-2m. The RGBD depth camera adjusts the depth measurement accuracy and image analysis algorithm parameters to focus on human target detection at a distance of 1.5m-2m and a range of 60 degrees.

[0091] Users can adjust the recognition distance on the settings page based on actual scenario requirements to meet personalized usage requirements in different scenarios. In close-range interactive scenarios (such as counter service), a shorter recognition distance can be set to improve accuracy; in open spaces (such as exhibition hall guidance), the distance range can be expanded to enhance coverage. Furthermore, a configurable non-interaction timeout mechanism prevents unnecessary device operation when no user needs it, improving the intelligence and user-friendliness of the service and enabling the device to better adapt to diverse commercial service scenarios.

[0092] In this embodiment, each sensor operates according to the adjusted parameters. Specifically, after the camera collects image data, it uses an improved face detection algorithm and human posture recognition algorithm for analysis. If a face or human posture is detected that meets the preset conditions and is within a distance of 1.5m and a range of 60 degrees, a trigger signal is sent. The ultrasonic sensor continuously monitors the reflected wave and triggers a recognition signal if it detects an approaching object within the set distance and angle range. The lidar scans to obtain point cloud data, analyzes the point cloud features to determine whether there is a target that meets human characteristics within the effective range, and triggers a signal if so. The RGBD depth camera obtains depth image and color image information, and combines the adjusted algorithm to determine whether there is a human target. If so, a signal is triggered. After receiving the signals from each sensor, they are processed according to the human recognition judgment logic. If one sensor is triggered, the voice interaction preparation state is entered. If all sensors are not triggered, data collection and analysis continue.

[0093] In one embodiment, a human body recognition method based on a multimodal sensor is provided that can be applied to an enterprise's access control device, such as Figure 5 As shown, the specific process includes:

[0094] The access control device is in the standby page, at which time advertisements, emoticons and other information can be displayed on the page;

[0095] When each multimodal sensor starts working, it can collect human body data and start multimodal human recognition. The access control device can be awakened by employees clicking the screen or speaking.

[0096] After waking up the access control device for the first time, it can play a welcome message and enter the sound reception logic program, standby page logic, and human body recognition logic at the same time;

[0097] Among them, in the sound receiving logic program, the access control device starts to receive sound and detects whether someone is speaking. If someone is speaking, it receives the sound and broadcasts the corresponding content, otherwise it continues to receive the sound;

[0098] In the human body recognition logic, it detects whether there is a human body. If not, it will count down for 10 seconds. If there is still no human body during the 10 seconds, the radio and the announcement will be turned off.

[0099] In the standby page logic, it is detected whether there is a conversation or operation. If not, the timer will be 7 seconds. If there is still no conversation or operation during 7 seconds, the standby page will be entered to display advertisements, emoticons and other information.

[0100] It should be understood that, although the various steps in the above flow chart are shown in sequence as indicated by the arrows, these steps are not necessarily performed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the above flow chart may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these sub-steps or stages is not necessarily to be performed in sequence, but can be performed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0101] In one embodiment, Figure 6 As shown, a human body recognition system based on a multimodal sensor is provided, including: a sensor data acquisition module 610, a system wake-up module 620, an instruction execution module 630 and an instruction detection module 640, wherein:

[0102] The sensor data acquisition module 610 is used to detect the start instruction in real time, control each sensor to perform data acquisition operations according to the start instruction, and obtain data from each sensor;

[0103] The system wake-up module 620 is used to analyze the data of each sensor and determine whether to enter the system wake-up state according to the analysis result. If the system enters the system wake-up state, the system enters the system standby interactive state;

[0104] The instruction execution module 630 is used to perform human body recognition based on the data from various sensors when the system is in a standby interactive state, and after successful human body recognition, receive input voice data and execute corresponding voice instructions based on the voice data;

[0105] The instruction detection module 640 is used to enter a standby state and detect a start instruction in real time if no input voice data is received within a threshold time period after human body recognition is successful.

[0106] In one embodiment, the sensor data acquisition module 610 is also used to monitor the ambient sound data in real time, perform real-time analysis on the ambient sound data, and obtain analysis results; determine whether a wake-up word appears in the ambient sound data based on the analysis results; if a wake-up word appears, continue to detect touch signals, and use the detected touch signals as startup instructions for each sensor to perform data acquisition operations; if no wake-up word appears, continue to monitor the ambient sound data.

[0107] In one embodiment, the sensor data acquisition module 610 is also used to collect gesture image information in real time through a depth camera or an infrared camera; perform gesture recognition on the gesture image information based on image recognition technology to obtain gesture recognition results; and control each sensor to perform data acquisition operations according to the gesture recognition results.

[0108] In one embodiment, each sensor includes a camera, an ultrasonic sensor, a lidar, and an RGBD depth camera; the sensor data acquisition module 610 is also used to obtain image data collected by the camera; obtain the reflected wave received after the ultrasonic sensor emits ultrasonic waves; obtain point cloud data collected by scanning the surrounding environment through the lidar; and obtain depth image and color image information obtained by the RGBD depth camera.

[0109] In one embodiment, the system wake-up module 620 is also used to analyze the image data, reflected wave, point cloud data, depth image and color image information respectively to obtain analysis results; if the analysis result is that the image data contains a face and the confidence level is higher than the set threshold; or the reflected wave meets the trigger condition; or the point cloud data contains human feature point cloud data; or the depth image and color image information contain human targets, then the system enters the wake-up state.

[0110] In one embodiment, the instruction execution module 630 is also used to, when the system is in standby interactive state, if the image data contains a human face and the confidence level is higher than a set threshold; or the reflected wave meets the trigger condition; or the point cloud data contains human feature point cloud data; or the depth image and color image information contain human targets, then human body recognition is successful and the input voice data is received.

[0111] In one embodiment, the system wake-up module 620 is further used to activate the sensor interface corresponding to each sensor if the system is in the wake-up state or the system standby interactive state; and to shut down the sensor interface corresponding to each sensor if the system is not in the wake-up state or the system standby interactive state.

[0112] In one embodiment, the sensor data acquisition module 610 is further used to obtain sensor parameter adjustment instructions and display a parameter setting interface according to the sensor parameter adjustment instructions; collect adjustment parameters through the parameter setting interface, and adjust the parameters of each sensor according to the adjustment parameters.

[0113] In one embodiment, a computer device is provided. The computer device may be an access control device, and its internal structure diagram may be as follows: Figure 7 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a human body recognition method based on a multimodal sensor is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad provided on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0114] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0115] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps of a human body recognition method based on a multimodal sensor when executing the computer program.

[0116] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of a human body recognition method based on a multimodal sensor are implemented.

[0117] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0118] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0119] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A human body recognition method based on a multimodal sensor, characterized in that: The method comprises: Detecting the start instruction in real time, controlling each sensor to perform data acquisition operations according to the start instruction, and acquiring data from each sensor; Analyze the sensor data and determine whether to enter the system awake state according to the analysis result. If the system awake state is entered, enter the system standby interactive state; In the standby interactive state of the system, human body recognition is performed according to the data of each sensor, and after successful human body recognition, input voice data is received and corresponding voice instructions are executed according to the voice data; If no voice data is received within a threshold period after human body recognition is successful, the system enters a standby state and detects the start command in real time.

2. The human body recognition method based on multimodal sensor according to claim 1, characterized in that: Real-time detection of start instructions, and control of each sensor to perform data acquisition operations according to the start instructions, including: monitoring environmental sound data in real time, analyzing the environmental sound data in real time, and obtaining analysis results; Determining whether a wake-up word appears in the ambient sound data according to the analysis result; If the wake-up word appears, the touch signal is continuously detected, and the detected touch signal is used as a start instruction for each sensor to perform data collection operations; If the wake-up word does not appear, continue to monitor the ambient sound data.

3. The human body recognition method based on multimodal sensor according to claim 1, characterized in that: Real-time detection of start instructions, and control of each sensor to perform data acquisition operations according to the start instructions, including: Collect gesture image information in real time through a depth camera or infrared camera; performing gesture recognition on the gesture image information based on image recognition technology to obtain a gesture recognition result; Control each sensor to perform data collection operations according to the gesture recognition result.

4. The human body recognition method based on multimodal sensor according to claim 1, characterized in that: Each of the sensors includes a camera, an ultrasonic sensor, a lidar, and an RGBD depth camera; Get various sensor data, including: Acquiring image data collected by the camera; Acquiring a reflected wave received after the ultrasonic sensor transmits the ultrasonic wave; Acquiring point cloud data collected by scanning the surrounding environment through the laser radar; and Acquire depth image and color image information acquired by the RGBD depth camera.

5. The human body recognition method based on multimodal sensor according to claim 4, characterized in that: Analyzing the sensor data and determining whether to enter the system wake-up state based on the analysis results includes: Analyzing the image data, reflected wave, point cloud data, depth image, and color image information respectively to obtain analysis results; If the analysis result is that the image data contains a human face and the confidence level is higher than a set threshold; or The reflected wave meets the trigger condition; or The point cloud data includes human body feature point cloud data; or If the depth image and color image information contain a human target, the system enters a wake-up state.

6. The human body recognition method based on multimodal sensor according to claim 4, characterized in that: In the standby interactive state of the system, human body recognition is performed according to the data of each sensor, and input voice data is received after the human body recognition is successful, including: In the standby interactive state of the system, if the image data contains a human face and the confidence level is higher than a set threshold; or The reflected wave meets the trigger condition; or The point cloud data includes human body feature point cloud data; or If the depth image and color image information contain a human target, the human body recognition is successful and the input voice data is received.

7. The human body recognition method based on multimodal sensor according to claim 1, characterized in that: The method further comprises: If the system is in a wake-up state or a standby interactive state, the sensor interface corresponding to each of the sensors is activated; If the system is not in the awake state or the standby interactive state, the sensor interface corresponding to each sensor is closed.

8. The human body recognition method based on multimodal sensor according to claim 1, characterized in that: The method further comprises: Obtaining a sensor parameter adjustment instruction and displaying a parameter setting interface according to the sensor parameter adjustment instruction; Adjustment parameters are collected through the parameter setting interface, and parameters of each of the sensors are adjusted according to the adjustment parameters.

9. A human body recognition system based on a multimodal sensor, characterized in that: The system comprises: The sensor data acquisition module is used to detect the start instruction in real time, control each sensor to perform data acquisition operations according to the start instruction, and obtain data from each sensor; A system wake-up module is used to analyze the data of each sensor and determine whether to enter the system wake-up state according to the analysis result. If the system enters the system wake-up state, the system enters the standby interactive state; An instruction execution module is used to perform human body recognition based on the data from each sensor when the system is in a standby interactive state, and receive input voice data after successful human body recognition, and execute corresponding voice instructions based on the voice data; The command detection module is used to enter a standby state and detect a start command in real time if no input voice data is received within a threshold time period after human body recognition is successful.

10. The human body recognition system based on multimodal sensors according to claim 9, characterized in that: The sensor data acquisition module is further configured to monitor ambient sound data in real time, perform real-time analysis on the ambient sound data, and obtain analysis results; and determine whether a wake-up word appears in the ambient sound data based on the analysis results; If the wake-up word appears, the touch signal is continuously detected, and the detected touch signal is used as a start instruction for each sensor to perform data collection operations; If the wake-up word does not appear, continue to monitor the ambient sound data.

Images