Posture recognition system and method and electronic equipment

CN119072724APending Publication Date: 2024-12-03BOE TECHNOLOGY GROUP CO LTD
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202380008550.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-03-31
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

The existing gesture recognition algorithm runs on the PC, resulting in a low frame rate, high latency, high computing power requirements, high system resource consumption, and poor user experience in communication between the camera and the PC.

Method used

Multiple depth cameras and independent processing modules (such as DSP) are used for gesture recognition. The recognition results are integrated through the processing module to reduce the computing power requirements and resource consumption of the control module. A soft synchronization scheme is used to avoid exposure interference between depth cameras.

Benefits of technology

It achieves low latency and high accuracy of gesture recognition, improves user experience, optimizes computing speed in DSP with limited hardware resources, and saves MCU resources.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119072724A_ABST
    Figure CN119072724A_ABST
Patent Text Reader

Abstract

The invention discloses a posture recognition system, a posture recognition method and electronic equipment, which are used for reducing the computing power demand and resource consumption of a control module and reducing recognition delay. The system comprises N depth cameras, N processing modules and a control module, the N depth cameras shoot target images from different positions and transmit the target images shot by the N depth cameras to the corresponding processing modules, and the target images comprise depth information of a target object; the N processing modules determine a depth map of a target object according to depth information contained in the target images received by the N processing modules respectively, carry out posture recognition on the target object in the target images in combination with the depth map of the target object to obtain N recognition results corresponding to the N depth cameras respectively, and send the N recognition results to the control module; and the control module fuses the N recognition results and determines the fusion result as the recognition result of the target image.
Need to check novelty before this filing date? Find Prior Art

Description

A gesture recognition system, method and electronic device Technical Field

[0001] The present disclosure relates to the field of image processing and recognition technology, and in particular to a gesture recognition system, method and electronic device. Background Art

[0002] Traditional human-computer interaction typically relies on touch. However, in areas like VR / AR and glasses-free 3D displays, sensors such as cameras are used to collect user gestures and other information. These gestures are then identified using relevant recognition and classification algorithms, which assign different semantic information to different gestures to enable different interactive functions. Existing gesture recognition algorithms mostly run on PCs.

[0003] Because the camera and PC communication cannot achieve a high frame rate, the overall latency of the gesture recognition algorithm is high. Furthermore, the gesture recognition algorithm running on the PC requires high computing power and consumes a lot of system resources, resulting in a long algorithmic process. This delay results in high overall gesture recognition latency and a poor user experience.

[0004] Summary of the Invention

[0005] The present disclosure provides a gesture recognition system, method, and electronic device for using a processing module to independently control a depth camera and perform gesture recognition, integrating the recognition results in a control module, reducing the computing power requirements and resource consumption of the control module, and reducing recognition delay.

[0006] In a first aspect, an embodiment of the present disclosure provides a gesture recognition system, the system comprising: N depth cameras, N processing modules, and a control module, wherein one processing module is configured to process a target image captured by a depth camera, where N is an integer greater than 1;

[0007] N depth cameras capture target images from different positions and transmit the captured target images to corresponding processing modules, wherein the target images contain depth information of the target object;

[0008] The N processing modules each determine a depth map of the target object based on the depth information contained in the target image they receive, and perform posture recognition on the target object in the target image in combination with the depth map of the target object, thereby obtaining N recognition results corresponding to the N depth cameras, and sending the N recognition results to the control module;

[0009] The control module fuses the N recognition results and determines the fusion result as the recognition result of the target image.

[0010] As an optional implementation manner, the processing module is specifically configured to:

[0011] Determine a depth map of the target object based on depth information contained in M ​​consecutive target image frames; wherein the phase information corresponding to the depth information contained in any two target image frames in the M frames is different; and M is determined based on the shooting parameters of the depth camera, and M is an integer greater than 0.

[0012] As an optional implementation manner, the processing module is specifically configured to:

[0013] The depth map of the target object and the target image are input into a gesture recognition model, and a recognition result of the target object is output; wherein the model parameters of the gesture recognition model for gesture recognition performed by different processing modules are different.

[0014] As an optional implementation, the processing module is further configured to utilize different kernels to perform operations on the depth map and gesture recognition respectively.

[0015] As an optional implementation, the control module includes a first submodule and a second submodule;

[0016] The first submodule is configured to receive the N recognition results and send the N recognition results to the second submodule;

[0017] The second submodule is configured to fuse the N recognition results to obtain a fusion result.

[0018] As an optional implementation manner, the first submodule is further configured to send any one or more of the following information to the processing module:

[0019] Firmware code, the firmware code is used to initialize the processing module;

[0020] Notification message, the notification message is used to notify the processing module to initialize the corresponding depth camera;

[0021] Model parameters, where the model parameters are used by the processing module to perform posture recognition on the received target image.

[0022] As an optional implementation manner, the first submodule is further configured to:

[0023] Using the processing module to send a pulse width modulation signal to the corresponding depth camera, and using the pulse width modulation signals of different depth cameras to control different depth cameras to perform exposure and shooting at intervals; or,

[0024] The processing module is used to adjust the register value of the corresponding depth camera, and the register values ​​of different depth cameras are used to control different depth cameras to perform exposure and shooting at intervals.

[0025] As an optional implementation manner, the first submodule is specifically configured as follows:

[0026] Determine the shooting interval duration of each of the N depth cameras and send it to the corresponding processing module, so that the processing module determines the register value of the corresponding depth camera based on the received shooting interval duration and sends it to the depth camera.

[0027] As an optional implementation manner, the second submodule is further configured to:

[0028] A separate sub-thread is used to receive N recognition results, and the N recognition results are fused to obtain a fusion result.

[0029] As an optional implementation manner, the second submodule is further configured to perform any one or more of the following:

[0030] Display N recognition results;

[0031] Display and store the fusion result;

[0032] A file containing model parameters input by a user is received, and the model parameters in the file are sent to a corresponding processing module through the first submodule.

[0033] As an optional implementation manner, the target object is a hand, the target image is a gesture image, and the recognition result is a gesture recognition result; the control module is further configured to:

[0034] Acquire the gesture recognition result, and use the gesture recognition result to perform a corresponding gesture interaction operation.

[0035] In a second aspect, an embodiment of the present disclosure provides a gesture recognition method, comprising:

[0036] N depth cameras capture target images from different positions and transmit the target images captured by each camera to corresponding processing modules, wherein the target images contain depth information of the target object; one processing module is used to process the target image captured by one depth camera, where N is an integer greater than 1;

[0037] The N processing modules each determine a depth map of the target object based on the depth information contained in the target image they receive, and perform posture recognition on the target object in the target image in combination with the depth map of the target object, thereby obtaining N recognition results corresponding to the N depth cameras, and sending the N recognition results to the control module;

[0038] The control module is used to fuse the N recognition results, and the fusion result is determined as the recognition result of the target image.

[0039] In a third aspect, an embodiment of the present disclosure further provides an electronic device, including a processor and a memory, wherein the memory is used to store a program executable by the processor, and the processor is used to read the program in the memory and perform the following steps:

[0040] N depth cameras capture target images from different positions and transmit the target images captured by each camera to corresponding processing modules, wherein the target images contain depth information of the target object; one processing module is used to process the target image captured by one depth camera, where N is an integer greater than 1;

[0041] The N processing modules each determine a depth map of the target object based on the depth information contained in the target image they receive, and perform posture recognition on the target object in the target image in combination with the depth map of the target object, thereby obtaining N recognition results corresponding to the N depth cameras, and sending the N recognition results to the control module;

[0042] The control module is used to fuse the N recognition results, and the fusion result is determined as the recognition result of the target image.

[0043] In a fourth aspect, an embodiment of the present disclosure further provides a computer storage medium on which a computer program is stored, which, when executed by a processor, is used to implement the steps of the method described in the first aspect above.

[0044] These and other aspects of the present disclosure will become more readily apparent from the following description of the embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0046] FIG1 is a schematic diagram of a gesture recognition system provided by an embodiment of the present disclosure;

[0047] FIG2 is a schematic diagram of an image flow of a TOF camera provided by an embodiment of the present disclosure;

[0048] FIG3 is a schematic diagram of the architecture of an identification system provided by an embodiment of the present disclosure;

[0049] FIG4 is a schematic diagram of the internal architecture of a DSP provided by an embodiment of the present disclosure;

[0050] FIG5 is a flowchart of gesture recognition of a DSP provided by an embodiment of the present disclosure;

[0051] FIG6 is a detailed flow chart of gesture recognition performed by a DSP according to an embodiment of the present disclosure;

[0052] 7A-7B are flowcharts of a method for implementing interval exposure by adjusting a register according to an embodiment of the present disclosure;

[0053] FIG8 is a control flow chart of an MCU provided by an embodiment of the present disclosure;

[0054] FIG9 is a flowchart of a PC execution according to an embodiment of the present disclosure;

[0055] FIG10 is a flowchart of an implementation of a gesture recognition method provided by an embodiment of the present disclosure;

[0056] FIG11 is a schematic diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0057] To make the objectives, technical solutions, and advantages of the present disclosure more clear, the present disclosure will be further described in detail below with reference to the accompanying drawings. It is apparent that the embodiments described are only a portion of the embodiments of the present disclosure, rather than all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present disclosure without creative effort are intended to fall within the scope of protection of the present disclosure.

[0058] In the embodiments of the present disclosure, the term "and / or" describes the association relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0059] The application scenarios described in the embodiments of the present disclosure are intended to more clearly illustrate the technical solutions of the embodiments of the present disclosure and do not constitute a limitation on the technical solutions provided by the embodiments of the present disclosure. Persons skilled in the art will appreciate that, as new application scenarios emerge, the technical solutions provided by the embodiments of the present disclosure will also be applicable to similar technical problems. In the description of the present disclosure, unless otherwise specified, "multiple" means two or more.

[0060] Traditional human-computer interaction typically relies on touch. However, in areas like VR / AR and glasses-free 3D displays, sensors such as cameras are used to collect user gestures and other information. These gestures are then identified using relevant recognition and classification algorithms, with different semantic information assigned to each gesture to enable different interactive functions. Existing gesture recognition algorithms mostly run on PCs. Because camera-PC communication cannot achieve a high frame rate, the overall latency of gesture recognition algorithms is high. Furthermore, gesture recognition algorithms running on PCs require high computing power and consume a lot of system resources, resulting in a long algorithmic process. This latency leads to high overall gesture recognition latency and a poor user experience.

[0061] Taking gesture recognition as an example, existing gesture recognition algorithms are mostly run on PCs. Due to communication issues between the camera and the PC, high frame rates cannot be achieved, resulting in high overall latency. Furthermore, gesture recognition algorithms running on PCs require high computing power and consume a lot of system resources, making the algorithm time-consuming. These two delays contribute to high overall gesture recognition latency and a poor user experience. Furthermore, for the bare-hand interaction scenarios provided by glasses-free 3D displays, using binocular cameras offers low accuracy and is ineffective on large screens.

[0062] In order to reduce the delay of posture recognition such as hand gesture recognition, the embodiment of the present disclosure provides a posture recognition system, which performs posture recognition for each depth camera through a separate processing module, implements the posture recognition algorithm through a separate processing module, and achieves low latency for overall recognition. In addition, based on multiple depth cameras shooting target images from different positions, it is possible to obtain information of target images at different angles, thereby improving recognition accuracy.

[0063] As shown in FIG1 , this embodiment provides a gesture recognition system, which includes: N depth cameras 100 , N processing modules 101 , and a control module 102 , wherein one processing module is configured to process a target image captured by a depth camera, where N is an integer greater than 1;

[0064] N depth cameras capture target images from different positions and transmit the captured target images to corresponding processing modules, wherein the target images contain depth information of the target object;

[0065] The N processing modules each determine a depth map of the target object based on the depth information contained in the target image they receive, and perform posture recognition on the target object in the target image in combination with the depth map of the target object, thereby obtaining N recognition results corresponding to the N depth cameras, and sending the N recognition results to the control module;

[0066] The control module fuses the N recognition results and determines the fusion result as the recognition result of the target image.

[0067] In some embodiments, the depth camera of this embodiment includes but is not limited to a ToF (Time of flight) camera. The ToF camera in this embodiment refers to a ToF sensor that emits infrared light or laser, where the generated light will bounce off any object and return to the sensor. Based on the time difference between the emission of the light and the time it is reflected by the object and returned to the sensor, the sensor can measure the distance between the object and the sensor. Taking gesture recognition as an example, the TOF camera is used to capture the position of the user's gesture in the stereoscopic display space, and the camera is used to capture gesture images to identify gesture actions, and finally the position and action of the gesture during the user interaction are obtained, thereby enabling interaction between bare hands and 3D content.

[0068] It should be noted that the depth camera in this embodiment is not limited to a TOF camera, and any camera that can collect depth information of a target object falls within the scope of the depth camera covered in this disclosure.

[0069] Optionally, the target object in this embodiment includes but is not limited to a person, a robot, an animal, etc. The target object may also be a finger, palm, hand, joint, arm, leg, foot, or other operable body part. This embodiment does not impose any further limitations on this.

[0070] Optionally, the processing modules in this embodiment include, but are not limited to, hardware devices with data processing capabilities, such as a DSP, and this embodiment does not impose any further limitations on this. In implementation, one processing module is used to process a target image captured by a depth camera, perform gesture recognition on the target image, and obtain a recognition result. Different processing modules are used to process target images captured by different depth cameras.

[0071] In some embodiments, the target object is a hand, the target image is a gesture image, and the recognition result is a gesture recognition result; the control module is further configured to:

[0072] Acquire the gesture recognition result, and use the gesture recognition result to perform a corresponding gesture interaction operation.

[0073] Optionally, when the control module includes a first submodule and a second submodule, and the second submodule includes a PC, the gesture recognition result is obtained from the shared memory storing the gesture recognition result using the unity application of the PC, and the gesture interaction operation is performed.

[0074] In some embodiments, each of the processing modules is specifically configured to:

[0075] Determining a depth map of the target object based on depth information contained in the received target image;

[0076] In combination with the depth map of the target object, posture recognition is performed on the target object in the target image to obtain a recognition result of the target object.

[0077] In practice, each frame of a target image captured by a depth camera contains depth information. A depth map of the target object can be calculated based on this depth information. In this embodiment, the depth map represents the distance of the target object relative to the depth camera. After obtaining the depth map and target image, posture recognition can be performed on the target object to determine its distance from the depth camera and its posture. This distance and posture can then be used to enable interaction in a stereoscopic display space.

[0078] In some embodiments, the processing module is specifically configured to:

[0079] Determine a depth map of the target object based on depth information contained in M ​​consecutive target image frames; wherein the phase information corresponding to the depth information contained in any two target image frames in the M frames is different; and M is determined based on the shooting parameters of the depth camera, and M is an integer greater than 0.

[0080] In implementation, since different depth information is represented by different phase information, depth information of multiple frames of target images is required to determine the depth map. Taking the depth camera as a TOF camera as an example, as shown in Figure 2, this embodiment provides an image flow diagram of a TOF camera, wherein each DCS (Differential Correlation Sample) is a phase map, representing a frame of target image, and calculating the depth map requires 4 phase maps, that is, 4 frames of target images are required to calculate the depth map of the target object.

[0081] In some embodiments, the processing module is specifically configured to:

[0082] The depth map of the target object and the target image are input into a gesture recognition model, and a recognition result of the target object is output; wherein the model parameters of the gesture recognition model for gesture recognition performed by different processing modules are different.

[0083] During implementation, the depth map and target image are simultaneously input into the gesture recognition module to obtain a recognition result for the target object. Optionally, the gesture recognition model in this embodiment includes, but is not limited to, a hand gesture recognition model or other models used for gesture recognition, which are not specifically limited in this embodiment. The gesture recognition model in this embodiment is placed in the processing module. Each processing module uses its own gesture recognition model to perform gesture recognition on the received target image. The model parameters of the gesture recognition models corresponding to different processing modules vary.

[0084] It should be noted that the posture recognition model and the posture recognition algorithm in this embodiment belong to the same concept. Similarly, the gesture recognition model, gesture recognition algorithm, and recognition algorithm all belong to the same concept and represent the same algorithm.

[0085] In some embodiments, the processing module is further configured to utilize different kernels to perform operations on the depth map and gesture recognition respectively.

[0086] In implementation, taking a DSP as an example, the DSP includes cores 0 and 1. Core 0 calculates depth information to generate a depth map, while core 1 performs gesture recognition to obtain the recognition result. This approach is used to allocate computing resources, optimize code flow, and increase computing speed within a DSP with limited hardware resources.

[0087] In some embodiments, the control module in this embodiment includes a first submodule and a second submodule;

[0088] The first submodule is configured to receive the N recognition results and send the N recognition results to the second submodule;

[0089] The second submodule is configured to fuse the N recognition results to obtain a fusion result.

[0090] Optionally, the first submodule in this embodiment includes but is not limited to a hardware device with data processing and control capabilities such as an MCU (microcontroller unit). The second submodule in this embodiment includes but is not limited to a host computer with an operating system such as a PC, terminal, or tablet computer.

[0091] As shown in Figure 3, this embodiment provides an architectural diagram of a recognition system, wherein the depth camera includes a TOF camera, the processing module includes a DSP, and the control module includes an MCU and a PC. Taking a 5-channel TOF camera as an example, each TOF camera is controlled by a separate DSP for processing the captured target image. The DSP is used to initialize the TOF camera and receive the target image sent by the TOF camera. After receiving the target image, it runs the gesture recognition algorithm. After the algorithm is completed, the recognition results obtained by the 5-channel DSP processing are transmitted to the MCU, and then sent by the MCU to the PC for fusion processing.

[0092] In some embodiments, the first submodule is further configured to send any one or more of the following information to the processing module:

[0093] Firmware code, the firmware code is used to initialize the processing module;

[0094] Notification message, the notification message is used to notify the processing module to initialize the corresponding depth camera;

[0095] Model parameters, where the model parameters are used by the processing module to perform posture recognition on the received target image.

[0096] In implementation, taking the MCU as the first submodule, the system initializes each DSP during startup. This involves sending firmware code, notification messages, and model parameters. The notification messages notify the DSPs to initialize the corresponding TOF cameras, while the model parameters are used by the DSPs for gesture recognition. During operation, the system receives recognition results from the DSPs and sends them to the PC.

[0097] In some embodiments, when the processing module is a DSP, since the DSP itself does not have a ROM and cannot store firmware code, the DSP cannot work independently and needs to be initialized by other control modules and send firmware code. In this embodiment, the MCU can control the DSP and initialize the DSP.

[0098] In some embodiments, five time-of-flight (TOF) cameras can be used, each placed in a different location. The gesture recognition model parameters in the DSP corresponding to each TOF camera are different. Due to limited MCU space, it is not possible to store excessive DSP firmware code. Therefore, in this embodiment, the same DSP firmware code can be used. The gesture recognition algorithm is optimized, the model parameters used in the algorithm are extracted, and a calling interface is added to the DSP. The MCU then distributes the different model parameters to the different TOF cameras.

[0099] In some embodiments, as shown in FIG4 , this embodiment also provides a schematic diagram of the internal architecture of a DSP, including two cores (Core 0 and Core 1). After receiving the firmware code from the MCU, the DSP initializes each module within the DSP. After initialization, both cores (Core 0 and Core 1) within the DSP enter an infinite loop to retrieve tasks from the task queue. If a task is retrieved, it is executed; otherwise, the next loop is entered.

[0100] Among them, the DSP receives MCU messages and the image data (including target images) sent by the TOF camera are triggered by interrupts. Simple calculations are performed in the interrupt handler, and complex calculations generate tasks and place them in the task queue.

[0101] DSP computing power allocation: The DSP interrupt processing core is Core 0. Because the gesture recognition algorithm is a relatively demanding task, it is independently processed in Core 1, while the depth information calculation algorithm is processed in Core 0. By allocating computing resources and optimizing the code flow within the limited DSP hardware resources, the self-developed algorithm is ensured to complete within a single frame time of 8.3ms at 120Hz.

[0102] As shown in FIG5 , taking the depth camera as a TOF camera, the processing module as a DSP, and the posture recognition as gesture recognition as an example, this embodiment provides a DSP gesture recognition process, and the specific implementation steps are as follows:

[0103] Step 500: The TOF camera sends a target image to the corresponding DSP;

[0104] In practice, the target image captured by the TOF camera is an IR image (infrared spectrum image), which contains depth information.

[0105] Step 501: When the DSP receives a frame of target image, it triggers an image interrupt processing program. In the image interrupt processing program, the DSP generates different processing tasks for different cores.

[0106] Step 502: Generate a depth information calculation task a every four frames of target image. After completing the calculation of task a using kernel 0, a depth map of the target object is obtained.

[0107] Step 503: A gesture recognition task b is generated for each frame of the target image. The task b is completed by calculation using kernel 1 in combination with the depth map of the target object.

[0108] During implementation, as shown in FIG6 , this embodiment also provides a detailed process for gesture recognition by a DSP. Before performing gesture recognition using core 1, the size of the target image can be converted to the size of the gesture recognition algorithm input. For example, a 12-bit target image can be converted to an 8-bit target image. The converted target image and the depth map are then input into the gesture recognition model, and the recognition result is output and sent to the MCU via the SPI (serial peripheral interface).

[0109] It should be noted that because depth calculation takes a long time (approximately 23ms), the depth map may not always be available when the converted target image and depth map are input into the gesture recognition model. The gesture recognition model can perform different recognition operations depending on whether the depth map of the target image is available.

[0110] In some embodiments, the first submodule in this embodiment is further configured to perform any of the following methods:

[0111] Method 1: Use the processing module to send a pulse width modulation signal to the corresponding depth camera, and use the pulse width modulation signals of different depth cameras to control the exposure and shooting intervals of different depth cameras.

[0112] Optionally, taking the first submodule as MCU and the processing module as DSP as an example, the MCU sends a pulse width modulation signal to each DSP, which is forwarded by each DSP to the corresponding depth camera, thereby controlling the interval exposure of each depth camera.

[0113] Method 2: Using the processing module to adjust the register value of the corresponding depth camera, and using the register values ​​of different depth cameras, control different depth cameras to perform exposure and shooting at intervals.

[0114] Optionally, taking the first submodule as MCU and the processing module as DSP as an example, the MCU calculates the register values ​​of each depth camera to control the interval exposure of different depth cameras. The MCU first sends each register value to the corresponding DSP, which is forwarded to the corresponding depth camera by each DSP, thereby controlling the interval exposure of each depth camera. The register value of the depth camera can control the shooting time and other information of the depth camera.

[0115] In implementation, taking the depth camera as a TOF camera as an example, since the TOF camera captures images by receiving the infrared light emitted by itself, for multiple TOF cameras, different TOF cameras cannot expose and shoot at the same time, otherwise they will interfere with each other, so a synchronization solution needs to be designed.

[0116] This embodiment designs the following synchronization solution to solve the synchronization problem between multiple TOF cameras and avoid interference between exposures of different TOF cameras:

[0117] Solution 1: Hard synchronization.

[0118] Optionally, the processing module is used to send a pulse width modulation signal to the corresponding depth camera, and the pulse width modulation signals of different depth cameras are used to control the different depth cameras to perform exposure and shooting at intervals.

[0119] Since each target image frame in a TOF camera is a DCS phase image, one phase image is represented here as a small frame. Since four phase images are required to calculate a depth map, four DCS phase images are represented as a large frame. In implementation, an additional PWM (pulse width modulation) signal must be provided to the TOF camera. Each time the TOF camera receives a PWM signal, it outputs four phase images, or four frames of target images, which are used to control the synchronization of each four frames of target images, or the synchronization of the large frame. For example, when the TOF camera is configured for 120 Hz, set the PWM signal to 30 Hz.

[0120] This solution is relatively stable, but each TOF camera requires a PWM signal, which places high demands on the hardware resources of the MCU that controls the TOF camera.

[0121] Solution 2: Soft synchronization.

[0122] Optionally, the processing module is used to adjust the register value of the corresponding depth camera, and the register values ​​of different depth cameras are used to control different depth cameras to perform exposure and shooting at intervals.

[0123] In some embodiments, the first submodule is specifically configured to:

[0124] Determine the shooting interval duration of each of the N depth cameras and send it to the corresponding processing module, so that the processing module determines the register value of the corresponding depth camera based on the received shooting interval duration and sends it to the depth camera.

[0125] In practice, for example, if the depth camera is a TOF camera and the processing module is a DSP, the TOF camera can adjust the interval of large frames (every 4 frames of target image) in real time by adjusting its own registers while continuously streaming. For example, at 120hz, the large frame interval is 8.3ms. If a 1ms delay is required relative to the current, simply change the interval to 9.3ms, run a large frame, and then change the large frame interval back to 8.3ms to achieve a 1ms delay. The register delivery time point can be selected in the following ways:

[0126] For example, during debugging, it was discovered that modifying the registers of this TOF camera would delay the effect of one large frame (containing four small frames). If the modification were made during the fourth small frame, the effect would be delayed by two large frames. To ensure time control, this embodiment can choose to update the TOF camera registers when receiving the target image of the first small frame, the second small frame, or the third small frame.

[0127] As shown in Figures 7A-7B, this embodiment provides a method for achieving interval exposure by adjusting registers. Taking the depth camera as a TOF camera, the processing module as a DSP, and the first submodule as an MCU as an example, the specific implementation process of this method is as follows:

[0128] Step 700: The MCU sends the delay time sync_interval_us relative to the reference time to each DSP via SPI, so that the DSP forwards it to the corresponding TOF camera.

[0129] Step 701: The DSP receives the delay time sent by the MCU and enters the SPI interrupt handler. In the SPI interrupt handler, the DSP obtains the current DSP system time, records it as the base time sync_base_us, and sets enable_sync to true.

[0130] Step 702: The DSP receives the target image sent by the TOF camera and enters the image interrupt handler. In the image interrupt handler, enable_sync is set to true, the time difference sync_tof_interval_us between the reference time and the current time is calculated, and the register value of the TOF camera is calculated based on the delay time and the time difference. Sync_count is reset to 1 and enable_sync is false. When enable_sync is false, the process ends.

[0131] As shown in FIG7B , in the implementation, in the image interrupt handler, the sync_count variable can be controlled to set the register to be updated when the target small frame image is received. For example, the register value is updated when the second small frame is received. If the current target image is the first small frame and the sync_count is 1, the register value is updated when the second small frame is received, and the sync_count is increased by 1 to become 2. If the current target image is the first small frame and the sync_count is 2, the register is restored to the 120hz configuration, and the sync_count is increased by 1 to become 3, and the process ends.

[0132] During implementation, multiple TOF cameras are synchronized using a soft synchronization method, which can save MCU resources and reduce the requirements for MCU selection.

[0133] In some embodiments, as shown in FIG8 , this embodiment provides a control flow of an MCU, as follows:

[0134] In the process of controlling the TOF camera through the DSP, the MCU first initializes its own module SPI, PWM, GPIO (General-purpose input / output, input and output pins), etc., and then allocates hardware resources (such as a set of SPI and control signals) for each TOF camera and saves them in the same structure to facilitate the distinction between the target images of different TOF cameras, as well as the firmware code, model parameters and other information of different DSPs. The TOF camera is reset by sending a notification message to the DSP, and the firmware code is sent to the DSP through SPI. The DSP initializes the TOF camera and model parameters by calling the DSP interface, and calls the DSP interface to send different model parameters for different TOF cameras. Finally, the TOF cameras are synchronized (using either hard synchronization or soft synchronization), and then enter normal operation. The recognition results of the DSP are read through SPI and then sent to the host computer (PC) through USB (serial port protocol).

[0135] In some embodiments, the second submodule is further configured to:

[0136] A separate sub-thread is used to receive N recognition results, and the N recognition results are fused to obtain a fusion result.

[0137] In practice, the second submodule can be a host computer, such as a PC. By placing the gesture recognition algorithm in the DSP, overall low latency is achieved. The overall hardware latency, from capturing a gesture to receiving the gesture recognition result from the host PC, is approximately 12ms. Adding the gesture recognition algorithm, latency can be reduced to less than 5ms.

[0138] In some embodiments, the second submodule is further configured to perform any one or more of the following:

[0139] a) Display N recognition results;

[0140] During implementation, using a PC as an example, the PC first initializes the main interface and serial port, then initializes data processing as a separate subthread to receive recognition results from N TOF cameras and fuse them using a fusion algorithm to produce a single fused result. On the main interface, an option is available to display all the recognition results from the N TOF cameras, primarily for debugging purposes.

[0141] b) displaying and storing the fusion result;

[0142] In implementation, the fusion results can be displayed in the main interface and stored in shared memory for easy access by other applications. For example, the Unity application can obtain gesture recognition results from the shared memory to perform various interactive operations.

[0143] c) receiving a file containing model parameters input by a user, and sending the model parameters in the file to a corresponding processing module through the first submodule.

[0144] In practice, in the main interface, you can import a file containing model parameters and send it to the MCU, which then sends it to the DSP. This allows you to debug the gesture recognition model online without having to repeatedly flash the MCU firmware.

[0145] As shown in Figure 9, this embodiment also provides a PC execution process, taking the depth camera as a TOF camera and N as 5 as an example, including initializing the main interface, initializing the serial port, and initializing the data processing sub-thread for receiving the recognition results of the 5-way TOF camera, and fusing them through the fusion algorithm to obtain a fusion result, displaying the recognition results of the 5-way TOF camera, displaying the fusion result, and storing the fusion result in the shared memory. It can also import a file containing model parameters and send it to the MCU, and then send it to the DSP through the MCU.

[0146] The gesture recognition system provided in this embodiment uses N depth cameras, each of which is independently controlled by a processing module (such as a DSP). A gesture recognition algorithm (such as a hand gesture recognition algorithm) runs in the processing module. The gesture recognition algorithm results are then sent to a second submodule (such as a PC) via a first submodule (such as an MCU). The PC aggregates the N recognition results and fuses them into a single fusion result, which is stored in shared memory. The Unity application then retrieves the fusion result from the shared memory, completing the entire interactive process. On the one hand, using multiple depth cameras to capture the gesture of the target object makes gesture recognition more accurate. On the other hand, by placing the gesture recognition algorithm in the DSP, overall low latency is achieved. Furthermore, within the DSP with limited hardware resources, computing resources are rationally allocated, code flow is optimized, and computing latency is further reduced. Finally, the synchronization problem between multiple depth cameras is solved to avoid interference between exposures. A soft synchronization method is used to save MCU resources and reduce the requirements for MCU selection.

[0147] Example 2: Based on the same inventive concept, the embodiment of the present disclosure also provides a gesture recognition method. The principle of solving the problem by this method is similar to that of the system. Therefore, the implementation of this method can refer to the implementation of the system, and the repeated parts will not be repeated.

[0148] As shown in FIG10 , the implementation process of a gesture recognition method provided by this embodiment is as follows:

[0149] Step 1000: Use N depth cameras to capture target images from different positions, and transmit the captured target images to corresponding processing modules;

[0150] Wherein the target image includes depth information of the target object; a processing module is used to process the target image taken by a depth camera, and N is an integer greater than 1;

[0151] Step 1001: Utilize N processing modules to determine a depth map of a target object based on depth information contained in each received target image, and perform posture recognition on the target object in the target image in combination with the depth map of the target object, thereby obtaining N recognition results corresponding to the N depth cameras, and sending the N recognition results to a control module;

[0152] Step 1002: Utilize the control module to fuse the N recognition results, and determine the fusion result as the recognition result of the target image.

[0153] As an optional implementation, each processing module determines the depth map of the target object in the following manner:

[0154] Determine a depth map of the target object based on depth information contained in M ​​consecutive target image frames; wherein the phase information corresponding to the depth information contained in any two target image frames in the M frames is different; and M is determined based on the shooting parameters of the depth camera, and M is an integer greater than 0.

[0155] As an optional implementation manner, performing posture recognition on the target object in the target image in combination with the depth map of the target object includes:

[0156] The depth map of the target object and the target image are input into a gesture recognition model, and a recognition result of the target object is output; wherein the model parameters of the gesture recognition model for gesture recognition performed by different processing modules are different.

[0157] As an optional implementation, it also includes:

[0158] Use different kernels to perform depth map and posture recognition operations respectively.

[0159] As an optional implementation, the control module includes a first submodule and a second submodule;

[0160] The first submodule is used to receive the N recognition results and send the N recognition results to the second submodule;

[0161] The second submodule is used to fuse the N recognition results to obtain a fusion result.

[0162] As an optional implementation manner, the method further includes using the first submodule to send any one or more of the following information to the processing module:

[0163] Firmware code, the firmware code is used to initialize the processing module;

[0164] Notification message, the notification message is used to notify the processing module to initialize the corresponding depth camera;

[0165] Model parameters, where the model parameters are used by the processing module to perform posture recognition on the received target image.

[0166] As an optional implementation, it also includes:

[0167] The first submodule uses the processing module to send a pulse width modulation signal to the corresponding depth camera, and uses the pulse width modulation signals of different depth cameras to control different depth cameras to perform exposure and shooting at intervals; or

[0168] The first submodule uses the processing module to adjust the register value of the corresponding depth camera, and uses the register values ​​of different depth cameras to control different depth cameras to perform exposure and shooting at intervals.

[0169] As an optional implementation, the first submodule determines the shooting interval duration of each of the N depth cameras and sends it to the corresponding processing module, so that the processing module determines the register value of the corresponding depth camera based on the received shooting interval duration and sends it to the depth camera.

[0170] As an optional implementation, the second submodule receives N recognition results using a separate sub-thread, and fuses the N recognition results to obtain a fusion result.

[0171] As an optional implementation, the method further includes using the second submodule to perform any one or more of the following:

[0172] Display N recognition results;

[0173] Display and store the fusion result;

[0174] A file containing model parameters input by a user is received, and the model parameters in the file are sent to a corresponding processing module through the first submodule.

[0175] As an optional implementation manner, the target object is a hand, the target image is a gesture image, and the recognition result is a gesture recognition result; further comprising:

[0176] Acquire the gesture recognition result, and use the gesture recognition result to perform a corresponding gesture interaction operation.

[0177] Example 3: Based on the same inventive concept, the embodiment of the present disclosure also provides an electronic device. The principle of solving the problem of the electronic device is similar to that of the system. Therefore, the implementation of the device can refer to the implementation of the system, and the repeated parts will not be repeated.

[0178] As shown in FIG11 , the electronic device includes a processor 1100 and a memory 1101 . The memory 1101 is used to store programs executable by the processor 1100 . The processor 1100 is used to read the programs in the memory 1101 and perform the following steps:

[0179] Utilize N depth cameras to capture target images from different positions, and transmit the captured target images to corresponding processing modules; wherein the target images contain depth information of the target object; a processing module is used to process the target image captured by one depth camera, where N is an integer greater than 1;

[0180] Utilize N processing modules to determine a depth map of the target object based on the depth information contained in the target image received by each module, and perform posture recognition on the target object in the target image in combination with the depth map of the target object, obtain N recognition results corresponding to the N depth cameras, and send the N recognition results to the control module;

[0181] The control module is used to fuse the N recognition results, and the fusion result is determined as the recognition result of the target image.

[0182] As an optional implementation, the processor 1100 is configured to use each processing module to determine a depth map of the target object in the following manner:

[0183] Determine a depth map of the target object based on depth information contained in M ​​consecutive target image frames; wherein the phase information corresponding to the depth information contained in any two target image frames in the M frames is different; and M is determined based on the shooting parameters of the depth camera, and M is an integer greater than 0.

[0184] As an optional implementation manner, the processor 1100 is specifically configured to:

[0185] The depth map of the target object and the target image are input into a gesture recognition model, and a recognition result of the target object is output; wherein the model parameters of the gesture recognition model for gesture recognition performed by different processing modules are different.

[0186] As an optional implementation manner, the processor 1100 is further configured to:

[0187] Use different kernels to perform depth map and posture recognition operations respectively.

[0188] As an optional implementation, the control module includes a first submodule and a second submodule;

[0189] The first submodule is used to receive the N recognition results and send the N recognition results to the second submodule;

[0190] The second submodule is used to fuse the N recognition results to obtain a fusion result.

[0191] As an optional implementation manner, the processor 1100 is further configured to use the first submodule to send any one or more of the following information to the processing module:

[0192] Firmware code, the firmware code is used to initialize the processing module;

[0193] Notification message, the notification message is used to notify the processing module to initialize the corresponding depth camera;

[0194] Model parameters, where the model parameters are used by the processing module to perform posture recognition on the received target image.

[0195] As an optional implementation manner, the processor 1100 is further configured to:

[0196] The first submodule uses the processing module to send a pulse width modulation signal to the corresponding depth camera, and uses the pulse width modulation signals of different depth cameras to control different depth cameras to perform exposure and shooting at intervals; or

[0197] The first submodule uses the processing module to adjust the register value of the corresponding depth camera, and uses the register values ​​of different depth cameras to control different depth cameras to perform exposure and shooting at intervals.

[0198] As an optional embodiment, the processor 1100 is specifically configured to use the first sub-module to determine the shooting interval duration of each of the N depth cameras and send it to the corresponding processing module, so that the processing module determines the register value of the corresponding depth camera based on the received shooting interval duration and sends it to the depth camera.

[0199] As an optional implementation, the second submodule receives N recognition results using a separate sub-thread, and fuses the N recognition results to obtain a fusion result.

[0200] As an optional implementation manner, the processor 1100 is further specifically configured to use the second sub-module to perform any one or more of the following:

[0201] Display N recognition results;

[0202] Display and store the fusion result;

[0203] A file containing model parameters input by a user is received, and the model parameters in the file are sent to a corresponding processing module through the first submodule.

[0204] As an optional implementation manner, the target object is a hand, the target image is a gesture image, and the recognition result is a gesture recognition result; the processor 1100 is further specifically configured to:

[0205] Acquire the gesture recognition result, and use the gesture recognition result to perform a corresponding gesture interaction operation.

[0206] Based on the same inventive concept, embodiments of the present disclosure provide a computer storage medium comprising computer program code. When the computer program code is executed on a computer, the computer executes any of the gesture recognition methods discussed above. Because the principles underlying the problem solved by the computer storage medium are similar to those of the gesture recognition method, the implementation of the computer storage medium can be referenced to the implementation of the method, and any repetitions will not be repeated here.

[0207] In a specific implementation process, computer storage media may include: Universal Serial Bus Flash Drive (USB), mobile hard disk, Read-Only Memory (ROM), Random Access Memory (RAM), magnetic disk or optical disk, and other storage media that can store program code.

[0208] Based on the same inventive concept, embodiments of the present disclosure further provide a computer program product, comprising: computer program code, which, when executed on a computer, causes the computer to perform any of the gesture recognition methods discussed above. Because the principles underlying the problems solved by the computer program products are similar to those of the gesture recognition methods, the implementation of the computer program products can be referenced to the implementation of the methods, and any repetitions will not be repeated.

[0209] The computer program product can employ any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0210] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage and optical storage, etc.) containing computer-usable program code.

[0211] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0212] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0213] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0214] Obviously, those skilled in the art may make various changes and modifications to the present disclosure without departing from the spirit and scope of the present disclosure. Thus, if these modifications and variations of the present disclosure fall within the scope of the claims of the present disclosure and their equivalents, the present disclosure is intended to include these modifications and variations.

Claims

1. A gesture recognition system, wherein: The system includes: N depth cameras, N processing modules and a control module, wherein one processing module is used to process a target image taken by a depth camera, and N is an integer greater than 1; N depth cameras capture target images from different positions, and transmit the captured target images to corresponding processing modules, wherein the target images contain depth information of the target object; The N processing modules respectively determine the depth map of the target object according to the depth information contained in the target image received by each of them, and perform posture recognition on the target object in the target image in combination with the depth map of the target object, so as to obtain N recognition results corresponding to the N depth cameras respectively, and send the N recognition results to the control module; The control module fuses the N recognition results and determines the fusion result as the recognition result of the target image.

2. The system according to claim 1, wherein: The processing module is specifically configured as follows: Determine a depth map of the target object according to depth information contained in M ​​consecutive target image frames; wherein phase information corresponding to the depth information contained in any two target image frames in the M frames is different; and M is determined according to shooting parameters of the depth camera, and M is an integer greater than 0.

3. The system according to claim 1, wherein: The processing module is specifically configured as follows: The depth map of the target object and the target image are input into a gesture recognition model, and a recognition result of the target object is output; wherein the model parameters of the gesture recognition model for gesture recognition performed by different processing modules are different.

4. The system according to claim 1, wherein: The processing module is further configured to utilize different cores to perform operations of depth map and gesture recognition respectively.

5. The system according to claim 1, wherein: The control module includes a first submodule and a second submodule; The first submodule is configured to receive the N recognition results and send the N recognition results to the second submodule; The second submodule is configured to fuse the N recognition results to obtain a fusion result.

6. The system according to claim 5, wherein: The first submodule is further configured to send any one or more of the following information to the processing module: A firmware code, wherein the firmware code is used to initialize the processing module; A notification message, wherein the notification message is used to notify the processing module to initialize the corresponding depth camera; Model parameters, wherein the model parameters are used by the processing module to perform posture recognition on the received target image.

7. The system according to claim 5, wherein: The first submodule is further configured as: Using the processing module to send a pulse width modulation signal to the corresponding depth camera, and using the pulse width modulation signals of different depth cameras to control different depth cameras to perform exposure and shooting at intervals; or, The processing module is used to adjust the register value of the corresponding depth camera, and the register values ​​of different depth cameras are used to control different depth cameras to perform exposure and shooting at intervals.

8. The system according to claim 7, wherein: The first submodule is specifically configured as follows: Determine the shooting interval duration of each of the N depth cameras and send it to the corresponding processing module, so that the processing module determines the register value of the corresponding depth camera based on the received shooting interval duration and sends it to the depth camera.

9. The system according to claim 5, wherein: The second submodule is further configured as follows: A separate sub-thread is used to receive N recognition results, and the N recognition results are fused to obtain a fusion result.

10. The system according to claim 5, wherein: The second submodule is further configured to perform any one or more of the following: Display N recognition results; Display and store the fusion result; A file containing model parameters input by a user is received, and the model parameters in the file are sent to a corresponding processing module through the first submodule.

11. The system according to any one of claims 1 to 10, wherein: The target object is a hand, the target image is a gesture image, and the recognition result is a gesture recognition result; the control module is further configured to: The gesture recognition result is obtained, and a corresponding gesture interaction operation is performed using the gesture recognition result.

12. A gesture recognition method, wherein: The method includes: Utilize N depth cameras to obtain target images from different positions, and transmit the target images captured by each of them to the corresponding processing module, wherein the target image contains the depth information of the target object; a processing module is used to process the target image captured by a depth camera, and N is an integer greater than 1; Using N processing modules to determine the depth map of the target object according to the depth information contained in the target images received by each of them, and performing posture recognition on the target object in the target image in combination with the depth map of the target object, to obtain N recognition results corresponding to the N depth cameras respectively, and send the N recognition results to the control module; The control module is used to fuse the N recognition results, and the fusion result is determined as the recognition result of the target image.

13. An electronic device, wherein: The electronic device comprises a processor and a memory, wherein the memory is used to store a program executable by the processor, and the processor is used to read the program in the memory and execute the steps of the method according to claim 12.

14. A computer storage medium having a computer program stored thereon, wherein: When the program is executed by a processor, the steps of the method as claimed in claim 12 are implemented.

Citation Information

Patent Citations

  • Unmarked motion capturing method and device based on multi-visual angle depth camera

    CN102800103A

  • Attitude recognition method and device, medium and computing equipment

    CN109919077A

  • Multi-camera figure association via pairwise matching in successive frames for immersive video

    CN115867938A

  • Multi-camera depth imaging

    US20140094307A1

  • Time-of-Flight (TOF) Assisted Structured Light Imaging

    US20150062558A1