Action recognition method, system, robot and storage medium

Abstract

This application provides an action recognition method, system, robot, and storage medium. The action recognition method includes: obtaining the current detection length corresponding to an associated object; obtaining the object length corresponding to the associated object; calculating the association distance based on the object length and the current detection length; and determining that the associated object performs a triggered action in response to the association distance satisfying the association distance condition. Determining whether the associated object performs a triggered action by using the association distance calculated from the current detection length and the object length of the associated object can improve the accuracy of action recognition.

Description

Technical Field

[0001] This application relates to the field of image recognition technology, and in particular to an action recognition method, system, robot, and storage medium. Background Technology

[0002] Tapping a designated area on the projection surface with a finger is one way for users to interact with the projection device. The device uses a depth camera to acquire images including the projection surface and the user's finger. After analyzing the coordinates of the user's finger joints in the images, it obtains the corresponding depth values ​​from the depth image. Based on the depth values, the device can determine that the user has performed a click action and can change the content on the projection surface in response to the user's click action.

[0003] Currently, when a user performs a click action on the projection surface, a shadow may be cast. The device may misinterpret the shadow portion of the captured image as the user's finger, leading to incorrect depth values ​​and incorrect determination of whether a click action was performed. Therefore, the device suffers from insufficient accuracy in action recognition. Summary of the Invention

[0004] In view of the above, it is necessary to provide an action recognition method, system, robot, and storage medium that can determine whether an associated object has executed a specified trigger action by recognizing the change in the current detection length corresponding to the associated object that performs the trigger action, thereby improving the accuracy of action recognition.

[0005] In a first aspect, embodiments of this application provide an action recognition method, the action recognition method comprising: obtaining the current detection length corresponding to an associated object; obtaining the object length corresponding to the associated object; calculating an association distance based on the object length corresponding to the associated object and the current detection length; and determining that the associated object performs a trigger action in response to the association distance satisfying an association distance condition.

[0006] Optionally, the action recognition method includes: obtaining a target distance corresponding to a target object, wherein the associated object moves in association with the target object, the target distance is the distance between the target object and the detection position, and the direction of the associated distance corresponds to the direction of the target distance; in response to the target distance satisfying the target distance condition, proceeding to the step of calculating the associated distance based on the object length corresponding to the associated object and the current detection length; in response to the target distance not satisfying the target distance condition, or in response to the associated distance not satisfying the associated distance condition, returning to the step of obtaining the target distance corresponding to the target object.

[0007] Optionally, obtaining the target distance corresponding to the target object includes: obtaining a depth image; identifying the target object in the depth image; obtaining the depth value corresponding to the target object in the depth image; and determining the depth value as the target distance corresponding to the target object.

[0008] Optionally, obtaining the object length corresponding to the associated object includes: monitoring the current detection length of the associated object; in response to the monitoring result of the current detection length satisfying the filtering conditions, filtering out the current detection length with the largest value from multiple current detection lengths, the filtering conditions including at least one of quantity conditions and time conditions; and determining the filtered current detection length as the object length of the associated object.

[0009] Optionally, the associated object includes a first node, a second node, and a third node, wherein the third node is located between the first node and the second node, and obtaining the current detection length corresponding to the associated object includes: obtaining the distance between the first node and the second node as a first length; calculating a second length based on the distance between the first node and the third node; and obtaining the current detection length from the first length and the second length.

[0010] Optionally, the target object includes a fourth node and a fifth node, and obtaining the object length corresponding to the associated object includes: obtaining the distance between the fourth node and the fifth node as a third length; and determining the third length as the object length corresponding to the associated object.

[0011] Optionally, the action recognition method includes: obtaining starting position information corresponding to the associated object, wherein the starting position information corresponds to the position of the associated object when the current object length is the object length; obtaining ending position information of the associated object, wherein the ending position information corresponds to the position of the associated object when the associated distance satisfies the associated distance condition; determining the movement distance of the associated object based on the starting position information and the ending position information; determining whether the movement distance satisfies the distance condition; in response to the movement distance satisfying the distance condition, determining that the associated object performs a first action; and in response to the movement distance not satisfying the distance condition, determining that the associated object performs a second action.

[0012] Secondly, embodiments of this application provide an action recognition system, including: a current detection length acquisition module, used to acquire the current detection length of an associated object; an object length module, used to acquire the object length corresponding to the associated object; a calculation module, used to calculate an association distance based on the object length corresponding to the associated object and the current detection length; and a judgment module, used to determine that the associated object performs a trigger action in response to the association distance satisfying the association distance condition.

[0013] Thirdly, embodiments of this application provide a robot, including: a robot body; a memory for storing a computer program; and a processor for executing the computer program stored in the memory. When the computer program is executed, the processor is used to execute the action recognition method as described in any of the preceding claims.

[0014] Fourthly, embodiments of this application provide a storage medium including a computer program that, when run on an electronic device, causes the electronic device to perform the action recognition method as described in any of the preceding claims.

[0015] The action recognition method, system, robot, and storage medium provided in this application can identify the current detection length of the associated object that is executing the triggering action, and calculate the association distance based on the object length corresponding to the associated object and the current detection length. This allows for the determination of whether the change in the current detection length causes the association distance to meet the association distance condition, and thus determines whether the associated object has executed the specified triggering action, effectively improving the accuracy of action recognition. Attached Figure Description

[0016] Figure 1 This is a system schematic diagram of the robot body in an embodiment of this application.

[0017] Figure 2 This is a schematic diagram of the robot body in an embodiment of this application.

[0018] Figure 3 This is a schematic diagram of the robot in an embodiment of this application.

[0019] Figure 4 This is a schematic diagram of the layout of the projection area and the detection range in an embodiment of this application.

[0020] Figure 5 This is a schematic diagram of the projection area in an embodiment of this application.

[0021] Figure 6 This is a schematic diagram of the target object and associated objects in the embodiments of this application.

[0022] Figure 7 This is another schematic diagram of the target object and associated objects in the embodiments of this application.

[0023] Figure 8 This is a flowchart of the action recognition method in the embodiments of this application.

[0024] Figure 9 This is another flowchart of the action recognition method in the embodiments of this application.

[0025] Figure 10 This is another flowchart of the action recognition method in the embodiments of this application.

[0026] Figure 11 This is another flowchart of the action recognition method in the embodiments of this application.

[0027] Figure 12 This is another flowchart of the action recognition method in the embodiments of this application.

[0028] Figure 13 This is another flowchart of the action recognition method in the embodiments of this application.

[0029] Figure 14 This is another flowchart of the action recognition method in the embodiments of this application.

[0030] Figure 15 This is a schematic diagram of the action recognition system in the embodiments of this application.

[0031] Explanation of key component symbols:

[0032]

[0033]

[0034] The following specific embodiments will further illustrate this application in conjunction with the above-described accompanying drawings. Detailed Implementation

[0035] The technical solutions in the implementation of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described implementation is only a part of the implementation of this application, and not all of the implementations.

[0036] In the following description, the use of suffixes such as "module," "component," or "unit" to denote parts is solely for the purpose of illustrative purposes and has no specific meaning in itself. Therefore, "module," "component," or "unit" may be used interchangeably.

[0037] Please see Figure 1 , Figure 1This is a schematic diagram of the hardware structure of a robot according to one embodiment of the present invention. The robot can be any of various types of robots, specifically including but not limited to at least one of wheeled robots, legged robots, tracked robots, crawling robots, worm-like robots, or swimming robots. For example, a robot can specifically be a legged robot, or a robot combining legged and wheeled features. Legged robots include monopodial robots, bipodial robots, or multipodial robots. A multipodial robot refers to a legged robot with three or more legs; for example, a multipodial robot can specifically be a quadrupedal robot. A robot is a machine capable of performing semi-autonomous or fully autonomous tasks. Robots are not limited to humanoid machine devices and can also include robots with configurations such as dog-shaped, horse-shaped, snake-shaped, or fish-shaped. For example, a robot can specifically be a quadrupedal robotic horse.

[0038] exist Figure 1 In the illustrated embodiment, the robot may include a robot body 100. The robot body 100 may include a mechanical unit 101, a communication unit 102, a sensing unit 103, an interface unit 104, a storage unit 105, a control module 110, a display unit 106, an input unit 107, and a power supply 111. The various components of the robot body 100 can be connected in any way, including wired or wireless connections. Those skilled in the art will understand that... Figure 1 The specific structure of the robot body 100 shown does not constitute a limitation on the robot body 100. The robot body 100 may include more or fewer parts than shown. Some parts are not essential components of the robot body 100 and can be omitted or combined as needed without changing the nature of the invention.

[0039] Please refer to the following: Figure 2 The following is combined with Figure 2 A detailed description of each component of the robot body 100 is provided below:

[0040] Mechanical unit 101 is the hardware of robot body 100. For example... Figure 1 As shown, the mechanical unit 101 may include a drive board 1011, a motor 1012, and a mechanical structure 1013, such as... Figure 2As shown, the mechanical structure 1013 may include a main body 1014, extendable legs 1015, and feet 1016. In other embodiments, the mechanical structure 1013 may also include an extendable robotic arm (not shown), a rotatable head structure 1017, a rocking tail structure 1018, a cargo-carrying structure 1019, a saddle structure 1020, etc. It should be noted that the various component modules of the mechanical unit 101 can be one or multiple, depending on the specific situation. For example, there may be four legs 1015, and each leg 1015 may be equipped with three motors 1012, resulting in a total of twelve motors 1012.

[0041] The communication unit 102 can be used for receiving and transmitting signals, and can also communicate with networks and other devices. For example, it can receive instructions from a remote control or other robots to move in a specific direction at a specific speed according to a specific gait, and then transmit these instructions to the control module 110 for processing. The communication unit 102 may include modules such as WiFi, 4G, 5G, Bluetooth, and infrared modules.

[0042] The sensing unit 103 is used to acquire information data about the environment surrounding the robot body 100 and to monitor parameter data of various components inside the robot body 100, and then sends this data to the control module 110. The sensing unit 103 may include various sensors, such as sensors for acquiring information about the surrounding environment: lidar (for remote object detection, distance determination, and / or velocity determination), millimeter-wave radar (for short-range object detection, distance determination, and / or velocity determination), cameras, infrared cameras, Global Navigation Satellite System (GNSS), etc. Sensors for monitoring various components inside the robot body 100 include: inertial measurement unit (IMU) (for measuring velocity, acceleration, and angular velocity values), foot sensors (for monitoring the position of the foot's contact point, foot posture, magnitude and direction of the contact force), and temperature sensors (for detecting component temperature). Other sensors that may be configured on the robot body 100, such as load sensors, touch sensors, motor angle sensors, and torque sensors, are not detailed here.

[0043] The interface unit 104 can be used to receive input from external devices (e.g., data, power, etc.) and transmit the received input to one or more components within the robot body 100, or it can be used to output to external devices (e.g., data, power, etc.). The interface unit 104 may include a power port, a data port (such as a USB port), a memory card port, a port for connecting a device with an identification module, an audio input / output (I / O) port, a video I / O port, etc.

[0044] Storage unit 105 is used to store software programs and various data. Storage unit 105 mainly includes a program storage area and a data storage area. The program storage area can store operating system programs, motion control programs, application programs (such as text editors), etc.; the data storage area can store data generated by the robot body 100 during use (such as various sensor data acquired by the sensing unit 103, log file data, etc.). Furthermore, storage unit 105 may include a high-speed random access memory 300, and may also include non-volatile memory 300, such as a disk storage device 300, flash memory, or other volatile solid-state memory 300.

[0045] The display unit 106 is used to display information input by the user or information provided to the user. The display unit 106 may include a display panel 1061, which may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.

[0046] Input unit 107 can be used to receive input numerical or character information. Specifically, input unit 107 may include touch panel 1071 and other input devices 1072. Touch panel 1071, also known as touch screen, can collect user touch operations (such as operations performed by the user using their palm, fingers, or suitable accessories on or near touch panel 1071) and drive corresponding connected devices according to a pre-set program. Touch panel 1071 may include touch detection device 1073 and touch controller 1074. Touch detection device 1073 detects the user's touch position and the signal generated by the touch operation, and transmits the signal to touch controller 1074; touch controller 1074 receives touch information from touch detection device 1073, converts it into touch point coordinates, sends it to control module 110, and can receive and execute commands from control module 110. In addition to touch panel 1071, input unit 107 may also include other input devices 1072. Specifically, other input devices 1072 may include, but are not limited to, one or more of the following: remote control handles, etc., without any specific limitation here.

[0047] Furthermore, the touch panel 1071 can cover the display panel 1061. When the touch panel 1071 detects a touch operation on or near it, it transmits the information to the control module 110 to determine the type of touch event. Subsequently, the control module 110 provides corresponding visual output on the display panel 1061 according to the type of touch event. Although in Figure 1In this embodiment, the touch panel 1071 and the display panel 1061 are two independent components that implement input and output functions respectively. However, in some embodiments, the touch panel 1071 and the display panel 1061 can be integrated to implement input and output functions. The specific implementation is not limited here.

[0048] The control module 110 is the control center of the robot body 100. It connects various components of the robot body 100 through various interfaces and lines. It controls the robot body 100 as a whole by running or executing software programs stored in the storage unit 105 and calling data stored in the storage unit 105.

[0049] Power supply 111 supplies power to various components. Power supply 111 may include a battery and a power control board. The power control board controls battery charging, discharging, and power consumption management. Figure 1 In the illustrated embodiment, power supply 111 is electrically connected to control module 110. In other embodiments, power supply 111 may also be electrically connected to sensing unit 103 (such as camera, radar, speaker, etc.) and motor 1012 respectively. It should be noted that each component may be connected to a different power supply 111, or be powered by the same power supply 111.

[0050] Based on the above embodiments, specifically, in some embodiments, a terminal device can be used to communicate with the robot body 100. When the terminal device communicates with the robot body 100, it can send instruction information to the robot body 100. The robot body 100 can receive the instruction information through the communication unit 102 and, upon receiving the instruction information, can transmit it to the control module 110, so that the control module 110 can process the instruction information to obtain the target speed value. The terminal device includes, but is not limited to, mobile phones, tablet computers, servers, personal computers, wearable smart devices, and other electrical devices with image capture capabilities.

[0051] The instruction information can be determined based on preset conditions. In one embodiment, the robot body 100 may include a sensing unit 103, which can generate instruction information based on the current environment of the robot body 100. The control module 110 can determine whether the current speed value of the robot body 100 meets the corresponding preset conditions based on the instruction information. If it does, the robot body 100 will maintain its current speed value and current gait; if it does not, a target speed value and a corresponding target gait will be determined based on the corresponding preset conditions, thereby controlling the robot body 100 to move at the target speed value and the corresponding target gait. Environmental sensors may include temperature sensors, air pressure sensors, vision sensors, and sound sensors. Instruction information may include temperature information, air pressure information, image information, and sound information. The communication method between the environmental sensors and the control module 110 can be wired or wireless. Wireless communication methods include, but are not limited to: wireless networks, mobile communication networks (3G, 4G, 5G, etc.), Bluetooth, and infrared.

[0052] Please see Figure 3 In some embodiments, the robot may further include a processor 200, a memory 300, a projection device 400, and a detection device 500. The processor 200, memory 300, projection device 400, detection device 500, and robot body 100 can establish a communication connection via a bus. The memory 300 is used to store computer programs. The processor 200 is used to execute the computer programs stored in the memory 300. When the processor 200 executes the computer programs stored in the memory 300, the processor 200 executes an action recognition method to identify the triggering action of the associated object 20. Figure 4 As shown, the projection device 400 projects an image onto the projection area 401. The detection device 500 works in conjunction with the processor 200 to detect the position of the target object 10. The processor 200 determines the position of the associated object 20 based on the position of the target object 10 and determines whether the associated object 20 has performed a trigger action. When the associated object 20 performs a trigger action, the processor 200 can control the projection device 400 to adjust the display content of the projection area 401. The movement of the target object 10 is associated with the movement of the associated object 20; the distance between the target object 10 and the detection position is associated with the distance between the associated object 20 and the detection position.

[0053] Please see Figure 3 and Figure 4In this embodiment, the relative relationship between the target object 10 and the associated object 20 is not specifically limited. For example, the target object 10 and the associated object 20 may be integrated, the associated object 20 may move synchronously when the target object 10 moves, and at least a portion of the associated object 20 may move relative to the target object 10. Specifically, the associated object 20 and the target object 10 may be a finger and the back of a hand, or the associated object 20 and the target object 10 may be a palm and an arm, respectively.

[0054] For example, the target object 10 and the associated object 20 can be separable. The target object 10 can selectively drive the associated object 20 to move synchronously, and the target object 10 can control the associated object 20 to move relative to the target object 10. Specifically, the target object 10 can be a hand, and the associated object 20 can be, but is not limited to, a pen, ruler, etc.

[0055] In this embodiment, the processor 200 may be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of the above-mentioned program.

[0056] The memory 300 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, a random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto. The memory 300 may exist independently or may be integrated with the processor 200; the embodiments of this application do not limit this.

[0057] It is understood that the memory 300 and the storage unit 105 can be set separately, or the storage unit 105 can be used as the memory 300 to store computer programs. This is not limited in the embodiments of this application.

[0058] It is understood that the processor 200 and the control module 110 can be set separately, or the control module 110 can be used as the processor 200 to execute the action recognition method. This is not limited in the embodiments of this application.

[0059] It is understood that the processor 200, memory 300, projection device 400 and detection device 500 can all be electrically connected to the power supply 111, and the power supply 111 can provide power to the processor 200, memory 300, projection device 400 and detection device 500.

[0060] In this embodiment, the installation locations of the processor 200, memory 300, projection device 400, and detection device 500 are not specifically limited. For example, the processor 200, memory 300, and projection device 400 can be fixedly installed at any part of the robot body 100, and the detection device 500 can be installed at a detection location outside the robot body 100. Alternatively, the processor 200, memory 300, projection device 400, and detection device 500 can all be fixedly installed at any part of the robot body 100. The specific method of fixed installation is possible, but not limited to, screw fixing, welding fixing, etc., and is not specifically limited in this embodiment.

[0061] It is understood that the projection device 400 can be an electronic device with at least one of the functions of two-dimensional image projection and three-dimensional image projection. For example, the projection device 400 can be, but is not limited to, a planar projection device, a holographic projection device, an augmented reality (AR) projection device, etc.

[0062] In this embodiment, the dimensions of the projection area 401 of the projection device 400 are not specifically limited. For example, the projection area 401 may be a planar area. Or, for example, the projection area 401 may be a three-dimensional space.

[0063] In this embodiment, the content and type of the image projected by the projection device 400 are not specifically limited. For example, the image projected by the projection device 400 may be a moving image. Alternatively, the image projected by the projection device 400 may be a static image. Or, the image projected by the projection device 400 may contain some moving images and some static images.

[0064] Please see Figures 3 to 5 It can be understood that the projection area 401 may include multiple display areas 402. Multiple display areas 402 can display different images. For example, some display areas 402 can display dynamic images, while others can display static icons. In this embodiment, the number, shape, and size of the display areas 402 are not limited.

[0065] It should be noted that the position of the projection area 401 can be preset before the motion recognition method runs. For example, the rated parameters of the projection device 400 may include the projection distance and the projection angle. The distance between the projection area 401 and the projection device 400 is equal to the projection distance in the rated parameters, and the extended area of ​​the projection area 401 corresponds to the projection angle in the rated parameters.

[0066] like Figure 5 As shown, the detection device 500 has a detection range 501, which may include the space between the detection position and the projection area 401. The detection device 500 can collect detection information within the detection range 501 and output the detection information to the processor 200. The processor 200 can determine whether the target object 10 has entered the detection range 501 based on the detection information, and determine the distance between the target object 10 that has entered the detection range 501 and the detection position.

[0067] It should be noted that the detection range 501 can be the space between the detection position and the projection area 401, and the positions of the detection position and the projection area 401 can be preset before the action recognition method runs. For example, the detection device 500 can be a device with a positioning module (not shown), and the processor 200 can be communicatively connected to the positioning module. The processor 200 can obtain the position information of the detection device 500 through the positioning module, that is, obtain the position information of the detection position. The position information can include, but is not limited to, the coordinate data of the detection position in a preset coordinate system. In the embodiments of this application, the preset coordinate system can be an existing public coordinate system, such as the geodetic coordinate system, latitude and longitude coordinate system, etc.

[0068] Similarly, the projection device 400 can be a device with a positioning module (not shown), and the processor 200 can obtain the position information of the projection device 400 through the positioning module. The principle by which the processor 200 obtains the position information of the projection device 400 is the same as or similar to the principle by which it obtains the position information of the detection device 500, and will not be described in detail here. At the same time, the processor 200 can determine the position information of the projection area 401 through the position information of the projection device 400 and the rated parameters of the projection device 400.

[0069] In this embodiment, the principle of the detection device 500 in acquiring detection information is not specifically limited, nor is the type of detection information specifically limited. For example, in a first scenario, the detection device 500 may be a depth camera, and the detection information may be a depth image. The detection device 500 may acquire a depth image of the detection range 501, and the acquired depth image includes a projection region 401. The detection device 500 is used to output the depth image to the processor 200. The processor 200 may identify the depth value at each location in the depth image. The processor 200 is used to identify the target object 10 in the depth image and obtain the depth value of the target object 10 at its location in the depth image, thereby determining the distance between the target object 10 and the detection location, i.e., determining the target distance.

[0070] For example, in the second scenario, the detection device 500 can be an image acquisition device, and the detection information can be a color image and / or an infrared image. The detection device 500 can acquire a color image and / or an infrared image at the location of the projection area 401, and output the acquired color image and / or infrared image to the processor 200. The processor 200 is used to identify the target object 10 in the color image and / or infrared image, and determine the distance between the target object 10 and the detection position based on the position, area, and other information of the target object 10 in the color image and / or infrared image, as well as the position information of the detection position, i.e., determine the target distance.

[0071] For example, in the third scenario, the detection device 500 can be a 3D LiDAR, and the detection information can be point cloud data. The detection device 500 can scan the projection area 401 and output the obtained point cloud data to the processor 200. The processor 200 is used to identify the target object 10 based on the point cloud data, and to determine the distance between the target object 10 and the detection position based on the point cloud data corresponding to the target object 10, that is, to determine the target distance.

[0072] It is understood that in the first and second scenarios described above, the projection area 401 can be a planar area. In the third scenario described above, the projection area 401 can be a three-dimensional space.

[0073] It should be noted that, in this embodiment, when a user needs to perform a trigger action through the associated object 20 to adjust the image of the projection area 401, the user needs to move the target object 10 and the associated object 20 to a position close to the projection area 401 (e.g., a planar area), or move the target object 10 and the associated object 20 into the projection area 401 (e.g., in three-dimensional space). That is, the target object 10 and the associated object 20 need to be moved into the detection range 501 of the detection device 500. Therefore, in this embodiment, when a user needs to perform a trigger action through the associated object 20, the detection information collected by the detection device 500 within the detection range 501 may include the information corresponding to the target object 10 and the associated object 20.

[0074] In this embodiment, the method by which the processor 200 identifies the target object 10 is not specifically limited. For example, the processor 200 can identify the target object 10 from the detection information collected by the detection device 500 using known, open-source software tools. Specifically, in some scenarios (such as the first and second scenarios described above), when the target object 10 is the user's palm, the processor 200 can identify the target object 10 in images (such as depth images, color images, and infrared images) using an open-source mediapipe model.

[0075] Please see Figures 3 to 6 Among them, such as Figure 6 As shown, processor 200 can recognize and analyze a user's hand using a mediapipe model, and determine the distribution of multiple nodes on the user's hand. These nodes may include 5 fingertip nodes and 16 knuckle points. Processor 200 can determine the distances between multiple nodes and detection locations (e.g., the depth values ​​corresponding to the nodes in a depth image), and it can also determine the straight-line connection distances between multiple nodes. For example, processor 200 can determine the sum of the straight-line distances between multiple adjacent nodes between node P9 and node P12, or determine the straight-line distance between node P9 and node P12, thereby determining the length of the user's middle finger.

[0076] It can be understood that the processor 200 can determine the target distance by determining the distance between a node on the target object 10 and the detection position, and then defining this distance as the distance between the target object 10 and the detection position. For example, when the target distance is the user's palm, the processor 200 can determine the distance between a knuckle on the hand and the detection position as the target distance. Specifically, as... Figure 4 and Figure 6As shown, when the target object 10 is the user's palm, and the processor 200 identifies the target object 10 in the image (e.g., depth image, color image, and infrared image) through the mediapipe model, the processor 200 can determine the distance between the metacarpophalangeal joint corresponding to the middle finger of the hand and the detection position as the target distance D1, that is, determine the distance between node P9 and the detection position as the target distance D1.

[0077] By determining the distance between the previous node of the target object 10 and the detection position as the target distance, the process of target distance recognition can be simplified and the efficiency of the processor 200 in acquiring the target distance can be improved.

[0078] It is understood that the processor 200 is also used to determine whether the target distance meets a preset target distance condition. In this embodiment, the content of the target distance condition is not specifically limited. For example, the target distance condition may be that the target distance is greater than a preset target threshold; when the target distance is greater than the preset target threshold, the processor 200 can determine that the target distance meets the target distance condition; otherwise, the processor 200 can determine that the target distance does not meet the target distance condition. Specifically, the target threshold may be, but is not limited to, 15 centimeters, and can be determined according to actual application requirements.

[0079] For example, the target distance condition could be that the target distance value is within a preset target range; when the target distance exceeds the preset target threshold, the processor 200 can determine that the target distance meets the target distance condition; otherwise, the processor 200 can determine that the target distance does not meet the target distance condition. Specifically, the target range can be, but is not limited to, a closed range of 10 cm to 16 cm, and can be determined according to actual application requirements.

[0080] It is understandable that when the target distance is determined not to meet the target distance condition, the detection device 500 continues to collect detection information of the target object 10 and outputs it to the processor 200. The processor 200 continues to determine the target distance corresponding to the target object 10 and continues to determine whether the target distance meets the target distance condition. It is understandable that the processor 200 can continuously monitor the target object 10 through the detection device 500, continuously monitor the target distance, and continuously determine whether the target object 10 meets the preset distance condition.

[0081] It is understood that when the target distance is determined to meet the target distance condition, the processor 200 is also used to obtain the current detection length corresponding to the associated object 20. In this embodiment of the application, when the processor 200 can identify the target object 10 through depth image, color image, infrared image or point cloud data, the processor 200 can also identify the associated object 20 through depth image, color image, infrared image or point cloud data.

[0082] In this embodiment, the method by which the processor 200 obtains the current detection length of the associated object 20 is not specifically limited. For example, in some scenarios (such as the first and second scenarios described above), the processor 200 can identify the associated object 20 in an image (e.g., a depth image, a color image, and an infrared image), and the length of the associated object 20 in the image is the current detection length at the current time point. The principle by which the processor 200 identifies the associated object 20 in the image is similar to or the same as the principle by which the processor 200 identifies the target object 10 in the image. The method by which the processor 200 identifies the associated object 20 in the image can be found in the description of the processor 200 identifying the target object 10 in the image above, and will not be repeated here.

[0083] Specifically, such as Figure 6 As shown, when the target object 10 is the user's palm, the associated object 20 is the user's index finger, and the processor 200 identifies the associated object 20 in the image using the mediapipe model, in the first implementation, the associated object 20 may include a first node and a second node, located at opposite ends of the associated object 20. For example, the first node may be node P5, and the second node may be node P8. The processor 200 can calculate a first length between the first node and the second node and determine this first length as the current detection length of the associated object 20. Specifically, when the first node is node P5 and the second node is node P8, the distance between the first node and the second node is the straight-line distance between node P5 and node P8.

[0084] In the second implementation, the associated object 20 may further include a third node. The third node is located between the first node and the second node. For example, the third node could be node P6. The processor 200 can calculate the straight-line distance between the third node and the first node, and multiply the calculated straight-line distance by a preset multiple to obtain a second length. The processor 200 uses the second length to determine the current detection length of the associated object 20. Specifically, when the first node is node P5 and the third node is node P6, the distance between the first node and the third node is the straight-line distance between node P5 and node P6.

[0085] It is understandable that the preset multiple corresponds to the proportion of the straight-line distance between the first node and the third node in the actual length of the associated object 20. Specifically, when the first node is node P5, the second node is node P8, and the third node is node P6, the preset multiple can be 2, which can be determined according to the actual application requirements.

[0086] It is understandable that the proportion of the straight-line distance between the first node and the third node in the actual length of the associated object 20 can be obtained by the staff through observation of experiments on obtaining the current detection length of the associated object 20. For example, the staff can determine through experimental observation that when the target object 10 is a palm and the associated object 20 is a finger (such as the index finger on the palm), twice the length of the proximal phalanx of the user's finger is approximately the same as the length of the user's finger.

[0087] It is understandable that when the target object 10 is the palm and the associated object 20 is a finger (e.g., the index finger on the palm), the user's fingers may bend in a relaxed state. For example, the part of the finger located on the proximal phalanx away from the palm may be relatively bent. In this case, the processor 200 determines the current detection length of the associated object 20 based on the images (e.g., depth images, color images, and infrared images) acquired by the detection device 500. This current detection length is the straight-line distance between the fingertip and the metacarpophalangeal joint when the user's finger is bent, which differs from the straight-line distance between the fingertip and the metacarpophalangeal joint when the user's finger is straight. To reduce the impact of the difference between the current detection length and the straight-line distance between the fingertip and the metacarpophalangeal joint when the user's finger is straight on the processor 200's recognition of the user's triggered action, twice the length of the proximal phalanx can be used as the current detection length of the associated object 200. That is, twice the second length can be used as the current detection length of the associated object 200.

[0088] In the third implementation, after the processor 200 calculates the first length and the second length, it compares the values ​​of the first length and the second length, and selects the one with the larger value as the current detection length of the associated object 20.

[0089] It is understandable that when the target object 10 is a palm and the associated object 20 is a finger (such as the index finger on the palm), if the first length is greater than the second length, the user's finger is more likely to be in a straight position, and the first length can be selected as the most accurate current detection length; if the second length is greater than the first length, the user's finger is more likely to be in a bent position, and the second length can be selected as the most accurate current detection length. In this way, the accuracy of the processor 200 in recognizing the user's actions can be improved.

[0090] For example, in other scenarios (such as the third scenario mentioned above), the processor 200 can identify the associated object 20 through point cloud data and calculate the current detection length of the associated object 20 through multiple point cloud data corresponding to the associated object 20.

[0091] It is understood that the current detection length of the associated object 20 can change over time. In this embodiment, the processor 200 can obtain multiple current detection lengths corresponding to the associated object 20 at multiple time points over time, that is, the processor 200 can continuously monitor the current detection length corresponding to the associated object 20.

[0092] It is understandable that after the processor 200 starts to obtain the current detection length of the associated object 20, the processor 200 is also used to obtain the object length corresponding to the associated object 20.

[0093] In the first implementation, the object length is the actual length of the associated object 20, and the object length value is a preset value. For example, when the associated object 20 is a ruler, the object length is the length of the long side of the ruler. As another example, when the associated object 20 is a user's finger, the object length is the length of the finger when it is fully extended, that is, the sum of the lengths of the knuckles between adjacent joints on the finger. The object length can be preset in the memory 300.

[0094] In the second implementation, the processor 200 selects one of the multiple currently detected lengths as the object length. This means that the selected object length is the currently detected length that is closest to the actual length of the associated object 20.

[0095] In this embodiment, the method by which the processor 200 determines the object length among multiple current detection lengths is not limited. For example, the processor 200 is used to determine whether the monitoring result of the current detection length meets the preset filtering conditions. When it is determined that the monitoring result of the current detection length meets the filtering conditions, the processor 200 can obtain the current detection length with the largest value among the multiple current detection lengths already obtained, and determine the current detection length with the largest value as the object length.

[0096] In some scenarios, the filtering criteria may include a quantity condition. The quantity condition can be that the number of current detection lengths monitored by the processor 200 reaches a quantity threshold. When the number of current detection lengths acquired by the processor 200 reaches the quantity threshold, the processor 200 determines that the monitoring result of the current detection length meets the quantity condition, i.e., the filtering condition is met; otherwise, the processor 200 determines that the monitoring result of the current detection length does not meet the quantity condition, i.e., the filtering condition is not met. Specifically, the quantity threshold can be, but is not limited to, 10, and can be determined according to actual application requirements.

[0097] In other scenarios, the filtering criteria may include time-related conditions. The time-related condition can be that the duration for which the processor 200 monitors the current detection length reaches a time threshold. When the processor 200 detects that the duration for which the current detection length reaches the time threshold, the processor 200 determines that the monitoring result of the current detection length meets the time condition, i.e., it meets the filtering criteria; otherwise, the processor 200 determines that the monitoring result of the current detection length does not meet the time condition, i.e., it does not meet the filtering criteria. Specifically, the time threshold can be, but is not limited to, 5 seconds, and can be determined according to the actual application requirements.

[0098] In other scenarios, the filtering criteria may include quantity and time conditions. When the processor 200 determines that the monitoring result of the current detection length meets both the quantity and time conditions, the processor 200 determines that the monitoring result of the current detection length meets the filtering criteria. When the processor 200 determines that the monitoring result of the current detection length does not meet at least one of the quantity or time conditions, the processor 200 determines that the monitoring result of the current detection length does not meet the filtering criteria.

[0099] It is understandable that the associated objects 20 for different users may be different. By filtering object lengths from multiple current detection lengths, the processor 200 can determine the object length only for the associated objects 20 corresponding to the multiple currently detected lengths. When another user moves the corresponding associated object 20 to the detection area, the processor 200 can first collect multiple current detection lengths for the other user's associated object 20, and then determine the object length from among the multiple current detection lengths corresponding to that user's associated object 20. In this way, the robot can recognize and respond to the trigger commands of multiple users.

[0100] In the third implementation, the target object 10 includes a fourth node and a fifth node. The processor 200 is used to determine the distance between the fourth node and the fifth node on the target object 10 as a third length, and uses the determined third length as the object length corresponding to the associated object 20.

[0101] In this embodiment of the application, the positions of the fourth and fifth nodes on the target object 10 are not limited. For example, as Figure 6 As shown, when the target object 10 is the user's palm and the associated object 20 is the user's index finger, the fourth node can be node P0 and the fifth node can be node P9. The processor 200 can determine the straight-line distance between node P0 and node P9 and use the determined straight-line distance as the object length (e.g., the sum of the straight-line distances of multiple adjacent nodes from node P5 to node P8).

[0102] It is understandable that the selection of the fourth and fifth nodes could be derived by staff through observations of experiments involving the acquisition of the object length of associated object 20. For example, experiments could show that the distance between the intercarpal joint of the user's wrist and the metacarpophalangeal joint corresponding to the user's middle finger is approximately equal to the length of the user's index finger.

[0103] It is understandable that different users can have target objects 10 with different shapes, sizes, and other related information. When the target object 10 and the associated object 20 are integrated, the size of the associated object 20 can correspond to the size of the target object 10. The processor 200 determines the distance between the fourth node and the fifth node on the target object 10 as the object length, which simplifies the process of obtaining the associated length. At the same time, by leveraging the correspondence between the associated object 20 and the target object 10, the processor 200 can separately obtain the object length of the associated object 20 for different users, thereby improving the accuracy of the processor 200 in obtaining the object length of different associated objects 20.

[0104] Please see Figures 3 to 7 In this embodiment, the processor 200 is further configured to calculate the association distance D2 based on the current detection length L1 and object length L2 corresponding to the associated object 20. The direction of the association distance D2 corresponds to the direction of the target distance D1. For example, both the association distance D2 and the target distance D1 can be the distance in the direction from the detection position toward the projection area 401.

[0105] It is understandable that when the processor 200 obtains the current detection length of the associated object 20 based on images (e.g., depth images, color images, and infrared images) acquired by the detection device 500, if one end of the associated object 20 moves toward or away from the projection area 401 (i.e., moves away from or toward the detection position), while the other end of the associated object 20 remains relatively stationary with respect to the target object 10, for example, when the target object 10 is a palm and the associated object 20 is a finger (e.g., the index finger on the palm), when the associated object 20 performs a click action on the projection area 401, the length of the associated object 20 in the image acquired by the detection device 500 will decrease, that is, the length of the projected image of the associated object 20 in the direction from the detection device 500 to the projection area 401 will decrease. The processor 200 can calculate the current detection length and object length corresponding to the associated object 20 according to the Pythagorean theorem, thereby obtaining the linear distance of the moving end of the associated object 20 in the direction from the detection position to the projection area 401, i.e., the association distance. Specifically, the projection area 401 can be a planar area on a wall. When the target object 10 is a palm and the associated object 20 is a finger (such as the index finger on the palm), when the user performs a click action with the finger, the associated distance is the distance that the fingertip moves in the direction perpendicular to the wall when the fingertip moves towards the wall.

[0106] In the embodiments of this application, the processor 200 can calculate the association distance according to the Pythagorean theorem. The association distance can be equal to the square root of the difference between the square of the object length corresponding to the associated object 20 and the current detection length corresponding to the associated object 20.

[0107] It is understood that the processor 200 is also used to determine whether the association distance meets the preset association distance conditions. In this embodiment of the application, the content of the association distance conditions is not specifically limited.

[0108] In some embodiments, the processor 200 is configured to determine whether the association distance meets a preset association distance condition based on the numerical value of the association distance. For example, the association distance condition may be that the association distance is greater than a preset association threshold; when the association distance is greater than the preset association threshold, the processor 200 may determine that the association distance meets the association distance condition; otherwise, the processor 200 may determine that the association distance does not meet the association distance condition. Specifically, the association threshold may be, but is not limited to, the difference between the distance between the projection area 401 and the detection position and the target distance, and then the difference between the target distance and a preset threshold value. The preset threshold value may be, but is not limited to, 1 cm, and can be determined according to actual application requirements.

[0109] It is understandable that in some scenarios (such as the first scenario mentioned above), the processor 200 can determine the depth value corresponding to the projection area 401 based on the depth image acquired by the detection device 500, thereby determining the distance between the projection area 401 and the detection position. In other scenarios (such as the second scenario mentioned above), the processor 200 can determine the position of the projection area 401 based on the position information and rated parameters of the projection device 400, and determine the position information of the detection position based on the position information of the detection device 500. Then, the distance between the projection area 401 and the detection position is calculated based on the determined position information of the projection area 401 and the position information of the detection position. In still other scenarios (such as the third scenario mentioned above), the processor 200 can determine the distance between the projection area 401 and the detection device 500 based on the point cloud data corresponding to the projection area 401 and the position information of the detection device 500.

[0110] For example, the association distance condition can be that the value of the association distance is within a preset association interval. When the association distance exceeds the preset association threshold, the processor 200 can determine that the association distance meets the association distance condition; otherwise, the processor 200 can determine that the association distance does not meet the association distance condition. Specifically, the association interval can be, but is not limited to, the closed interval between the distance between the projection area 401 and the detection position minus the target distance, and the difference after subtracting a preset threshold value (e.g., 1 cm), and the distance between the projection area 401 and the detection position. The specific interval can be determined according to the actual application requirements.

[0111] In other embodiments, the processor 200 calculates the result distance using the association distance and then determines whether the result distance is greater than a preset association distance or whether it is within a preset result interval. If the result distance is greater than a preset result threshold or is within a preset result interval, the processor 200 can determine that the association distance meets the association distance condition; otherwise, the processor 200 can determine that the association distance does not meet the association distance condition.

[0112] like Figure 4 and Figure 7 As shown, in some embodiments, the resulting distance D3 can be equal to the sum of the association distance D2 and the target distance D1. It can be understood that since the associated object 20 and the target object 10 move in association, the distance between the target object 10 and the detection position can indicate the distance between the detection position and the end of the associated object 20 that remains relatively stationary compared to the target object 10 when the associated object 20 performs the triggering action. That is, the target distance D1 can indicate the distance between the detection position and the end of the associated object 20 that remains relatively stationary compared to the target object 10 when the associated object 20 performs the triggering action. Simultaneously, the resulting distance D3 can indicate the distance between the detection position and the moving end of the associated object 20 when the associated object 20 performs the triggering action.

[0113] Specifically, when the processor 200 calculates the result distance through the correlation distance and then determines whether the result distance is greater than a preset result threshold, thereby determining whether the correlation distance meets the correlation distance condition, the result threshold can be, but is not limited to, the difference between the distance between the projection area 401 and the detection position minus 1 cm. It can be determined according to the actual application requirements.

[0114] Specifically, when the processor 200 calculates the result distance through the correlation distance and then determines whether the result distance is within a preset result interval, thereby determining whether the correlation distance meets the correlation distance condition, the result interval can be, but is not limited to, the closed interval between the difference between the distance between the projection area 401 and the detection position minus 1 cm and the distance between the projection area 401 and the detection position. The specific interval can be determined according to the actual application requirements.

[0115] It is understandable that when the correlation distance is determined not to meet the correlation distance condition, the detection device 500 continues to collect detection information and outputs it to the processor 200. In some scenarios, the processor 200 continues to determine the target distance corresponding to the target object 10 and whether the target distance meets the target distance condition. In other scenarios, the processor 200 continues to obtain the current detection length of the correlated object 20, continues to calculate the correlation distance based on the current detection length and object length corresponding to the correlated object 20, and continues to determine whether the correlation distance meets the correlation distance condition. It is understandable that the processor 200 continuously performs the steps of calculating the correlation distance and determining whether the correlation distance meets the correlation distance condition.

[0116] It is understandable that when the processor 200 determines that the association distance meets the association distance condition, the processor 200 can determine that the user has performed a trigger action through the association object 20. When the processor 200 determines that the association distance meets the association distance condition, the processor 200 controls the projection device 400 to adjust the projected image, so that the projected image changes in response to the trigger action of the association object 20, allowing the user to interact with the projection device 400 through the trigger action of the association object 20, that is, to interact with the robot through the trigger action of the association object 20.

[0117] It is understood that trigger actions can include swipe actions and click actions. In some embodiments, after the processor 200 determines that the associated object 20 has performed a trigger action, the processor 200 is also used to determine the type of trigger action performed by the associated object 20.

[0118] In this embodiment, the method by which the processor 200 determines the type of triggering action performed by the associated object 20 is not specifically limited. For example, the processor 200 can determine the type of triggering action performed by the processor 200 based on the distance between the start and end positions of the moving end on the associated object 20 when a click or swipe action is performed. For instance, the processor 200 is used to obtain the start and end position information of the associated object 20 through detection information, and calculate the movement distance of a preset node based on the start and end position information. The processor 200 is used to determine whether the movement distance meets a preset movement distance condition. If the movement distance meets the movement distance condition, the processor 200 determines that the triggering action is a swipe action; if the movement distance does not meet the movement distance condition, the processor 200 determines that the triggering action is a click action.

[0119] Understandable, such as Figure 6As shown, the start position information indicates the start time when the triggering action begins to be executed on the associated object 20, and the position of the preset node (e.g., node P8) on the associated object 20 (e.g., index finger) in the detection information (e.g., depth image), that is, the position of the preset node of the associated object 20 when the current detection length of the associated object 20 is the object length. The end position information indicates the end time when the triggering action is executed on the associated object 20, and the position of the preset node (e.g., node P8) on the associated object 20 (e.g., index finger) in the detection information (e.g., depth image), that is, the position of the preset node of the associated object 20 when the association distance meets the association distance condition.

[0120] In this embodiment, the form and content of the start and end position information are not limited. For example, the processor 200 can establish a detection coordinate system based on the detected information, where both the start and end position information can be the coordinate data of a preset node in the detection coordinate system. Specifically, in some scenarios (such as the first and second scenarios mentioned above), the detected information is an image (e.g., a depth image, a color image, and an infrared image). The processor 200 can establish a two-dimensional coordinate system in the image. For example, it can select an adjacent long side and a short side on a rectangular image, determine the extensions of the long side and the short side as the two coordinate axes of the coordinate system, and determine the intersection of the long and short sides as the origin of the coordinate system to establish the detection coordinate system. In this way, the processor 200 can determine the coordinate data of the preset node in the detection coordinate system as the end and start position information of the preset node.

[0121] Specifically, in other scenarios (such as the third scenario mentioned above), the detection information is point cloud data in three-dimensional space. The processor 200 can establish a three-dimensional detection coordinate system based on the data obtained by the detection device 500 (such as a three-dimensional LiDAR). The processor 200 can determine the point cloud data of the preset node as the end position information and start position information of the preset node.

[0122] It is understandable that the processor 200 can calculate the distance between two points in the coordinate system corresponding to the two positions based on the coordinate data in the starting position information and the ending position information, and use the calculated distance as the movement distance.

[0123] In this embodiment, the content of the movement distance condition is not specifically limited. For example, the movement distance condition may be that the movement distance is greater than a preset movement threshold; when the movement distance is greater than the preset movement threshold, the processor 200 can determine that the movement distance meets the movement distance condition; otherwise, the processor 200 can determine that the movement distance does not meet the movement distance condition. Specifically, the movement threshold may be, but is not limited to, 2 centimeters, and can be determined according to actual application requirements.

[0124] It is understood that when the associated object 20 performs a click or swipe action, the end of the associated object 20 corresponding to the preset node (e.g., the fingertip) will move away from the detection position. However, the movement distance of the preset node when the associated object 20 performs a click action is less than the movement distance when the associated object 20 performs a swipe action. The processor 200 can determine whether the end of the associated object 20 corresponding to the preset node has moved in any direction other than the direction relative to the detection position and the projection area 401 by determining whether the movement distance of the preset point is greater than a movement threshold. If it exists, it can be determined that the associated object 20 has performed a swipe action; if it does not exist, it can be determined that the associated object 20 has performed a click action.

[0125] like Figure 5 As shown, in some embodiments, the processor 200 is further configured to, after determining the type of triggering action performed by the associated object 20, adjust the image projected by the projection area 401 according to the display area 402 corresponding to the triggering action and the starting position information.

[0126] It is understandable that when a user needs to interact with the image in the designated display area 402 within the projection area 401, the user can point the end of the associated object 20 corresponding to the preset node (e.g., the fingertip) to the designated display area 402 before executing the trigger action. The processor 200 can determine the display area 402 where the preset node is located based on the position of the end of the preset node when the associated object 20 starts executing the trigger action, i.e., through the initial position information, and control the projection device 400 to change the image projected in part or all of the display area 402 within the projection area 401 according to the type of trigger action (e.g., a click action or a swipe action).

[0127] It is understood that when the associated object 20 performs different triggering actions on the same display area 402, the display area 402 in the projection area 401 where the image changes can be different, and the content of the changed image can be different. Similarly, when the associated object 20 performs the same triggering action on different display areas 402, the display area 402 in the projection area 401 where the image changes can be different, and the content of the changed image can be different. The different display areas 402 and the different triggering actions corresponding to the different display areas 402 and the changed image content can be preset by the staff.

[0128] The following is combined Figures 1 to 7 Taking the target object 10 as the palm, the associated object 20 as the fingers on the palm (e.g., the index finger), and the detection device 500 as a depth camera as an example, the principle of the robot recognizing the trigger action of the associated object 20 is explained in detail.

[0129] First, the projection device 400 projects an image into the projection area 401. The detection device 500 acquires a depth image of the detection range 501 and outputs detection information to the processor 200. The processor 200 can identify the target object 10 and associated object 20 in the depth image using a mediapipe model. The processor 200 obtains the depth value of the node above the target object 10 (e.g., node P9) as the distance between the target object 10 and the detection position, i.e., the target distance corresponding to the target object 10. After obtaining the target distance, the processor 200 determines whether the target distance meets the target distance condition. If the target distance does not meet the target distance condition, the processor 200 continues to acquire the target distance and continues to determine whether the target distance meets the target distance condition. If the target distance meets the target distance condition, the processor 200 begins to acquire the current detection length of the associated object 20. The processor 200 can determine the larger of a first distance (e.g., the distance between nodes P5 and P8) and a second distance (e.g., twice the distance between nodes P5 and P6) as the current detection length.

[0130] Next, processor 200 determines whether the monitoring result of the current detection length meets the filtering conditions (e.g., time and quantity conditions). If the monitoring result of the current detection length does not meet the filtering conditions, processor 200 continues to acquire the current detection length and continues to determine whether the monitoring result of the current detection length meets the filtering conditions. If the monitoring result of the current detection length meets the filtering conditions, processor 200 acquires the object length corresponding to the associated object 20. Processor 200 can determine the largest current detection length from the multiple monitored current detection lengths as the object length.

[0131] Then, the processor 200 can calculate the association distance based on the object length and the current detection length, and determine whether the association distance meets the association distance conditions (for example, first calculate the result distance using the association distance and the target distance, and then determine whether the result distance is greater than a preset result threshold). When the processor 200 determines that the associated object 20 does not meet the association distance conditions, the processor 200 can continue to acquire the current detection length, continue to calculate the association distance, and continue to determine whether the association distance meets the association distance conditions. When the processor 200 determines that the association distance meets the association distance conditions, the processor 200 can determine that the associated object 20 has executed a trigger action. The processor 200 can continue to acquire the start position information and end position information of the preset node of the associated object 20, calculate and determine the movement distance of the preset node based on the start position information and end position information, and determine the type of trigger action executed by the associated object 20 based on whether the movement distance meets the movement distance conditions. The processor 200 can change the projected image at a specified position in the projection area 401 according to the type of trigger action and the display area 402 corresponding to the start position information.

[0132] Obviously, in this embodiment, the processor 200 can identify the change in the projected length of the associated object 20 after its movement (i.e., the change in the current detected length of the associated object 20 from equal to the object length to less than the object length), and calculate the association distance based on the object length corresponding to the associated object 20 and the current detected length. The processor can then determine whether the associated object 20 has performed a triggering action based on the value of the association distance. This avoids the situation where the processor 200 misidentifies the shadow of the associated object 20 on the projection area 401 as the associated object 20 when identifying it in the depth image and determining whether the associated object 20 has performed a triggering action based on its depth value, thus obtaining an incorrect depth value for determining whether the associated object 20 has performed a triggering action. This improves the accuracy of identifying the associated object 20 and the accuracy of determining whether the associated object 20 has performed a triggering action, thereby improving the accuracy of action recognition.

[0133] Please see Figure 8 , Figure 8 A schematic diagram of an action recognition method provided in an embodiment of this application is shown. The action recognition method can be applied to the processor 200 described in any of the above embodiments. Figure 8 As shown, action recognition methods may include:

[0134] Step S81: Obtain the target distance corresponding to target object 10.

[0135] In step S81, the processor 200 can identify the target object 10 from the detection information (e.g., depth image) output by the detection device 500, and determine the distance between the target object 10 and the detection position as the target distance. For details, please refer to [link to relevant documentation]. Figures 3 to 7 The details and related descriptions will not be repeated here.

[0136] Please see Figure 9 In some embodiments, when the detection information is a depth image, the method for obtaining the target distance corresponding to the target object 10 may include the following steps S91 to S94:

[0137] Step S91: Obtain the depth image.

[0138] In step S91, the detection device 500 (e.g., a depth camera device) can acquire a depth image of the detection range 501, and the processor 200 can acquire the depth image acquired by the detection device 500.

[0139] Step S92: Identify the target object 10 in the depth image.

[0140] In step S92, the processor 200 can identify the target object 10 (e.g., a hand) in the depth image using the mediapipe model. (See details...) Figures 3 to 7 The details and related descriptions will not be repeated here.

[0141] Step S93: Obtain the depth value of the target object 10 in the depth image.

[0142] In step S93, the processor 200 can obtain the depth value of the previous node of the target object 10 as the depth value of the target object 10 in the depth image. For details, please refer to [link to relevant documentation]. Figures 3 to 7 The details and related descriptions will not be repeated here.

[0143] Step S94: Use the obtained depth value as the target distance corresponding to the target object 10.

[0144] It can be understood that the depth value obtained by the processor 200 is the distance between the target object 10 and the detection position, so the depth value can be used as the target distance corresponding to the target object 10.

[0145] Please continue reading. Figure 8 Step S82: Determine whether the target distance meets the target distance condition.

[0146] It is understandable that, in response to the target distance not meeting the target distance condition, the process returns to step S81 to reacquire the target distance corresponding to the target object 10. In response to the target distance meeting the target distance condition, the process proceeds to step S83.

[0147] It should be noted that, in the embodiments of this application, "in response to" is used to instruct the processor 200 to execute the specified steps only after the specified prerequisites are met (e.g., the target distance meets the target distance condition).

[0148] In step S82, the processor 200 can determine whether the target distance meets the target distance range based on whether the target distance is greater than the target threshold or whether it is within the target interval. For details, please refer to [link / reference needed]. Figures 3 to 7 The details and related descriptions will not be repeated here.

[0149] Step S83: Obtain the current detection length corresponding to the associated object 20.

[0150] In step S83, the processor 200 can identify the associated object 20 in the detection information (e.g., a depth image) and determine the current detection length of the associated object 20, as detailed in [reference needed]. Figures 3 to 7 The details and related descriptions will not be repeated here.

[0151] It is understandable that when the target distance corresponding to the target object 10 is determined to meet the target distance condition, it can be determined that there is a possibility that the user may execute a triggering action. The processor 200 can then begin to obtain the current detection length corresponding to the associated object 20 and determine whether the user should execute the triggering action through the associated object 20 based on the current detection length. The processor 200 can determine the current detection length only when the target meets the target distance condition, which can reduce the processing power consumption of the processor 200.

[0152] Please see Figure 10 In some embodiments, the method for obtaining the current detection length corresponding to the associated object 20 may include the following steps S101 to S103:

[0153] Step S101: Obtain the distance between the first node and the second node as the first length.

[0154] In step S101, the processor 200 can obtain the straight-line distance between the first node and the second node as the first length, as detailed in [reference needed]. Figures 3 to 7 The details and related descriptions will not be repeated here.

[0155] Among them, such as Figure 6 As shown, when the target object 10 is a palm and the associated object 20 is the index finger on the palm, the first node can be node P5 and the second node can be node P8.

[0156] Step S102: Calculate the second length based on the distance between the first node and the third node.

[0157] In step S102, the third node is located between the first node and the second node. The processor 200 can obtain the straight-line distance between the first node and the third node, and multiply the obtained straight-line distance by a preset multiple to obtain the second length. For details, please refer to [link / reference needed]. Figures 3 to 7 The details and related descriptions will not be repeated here. The preset multiple can be, but is not limited to, 2.

[0158] Among them, such as Figure 6 As shown, when the target object 10 is a palm and the associated object 20 is the index finger on the palm, the first node can be the P5 node, the second node can be the P8 node, and the third node can be the P6 node.

[0159] Step S103: Obtain the current detection length from the first length and the second length.

[0160] In step S103, the processor 200 can determine the larger of the first length and the second length as the current detection length corresponding to the associated object 20.

[0161] It is understandable that obtaining the length with the larger value between the first length and the second length as the current detection length can reduce the impact of bending of some parts of the associated object 20 on the accuracy of the current detection length, improve the accuracy of obtaining the current detection length, and improve the accuracy of determining whether the associated object 20 has executed a trigger action.

[0162] Please continue reading. Figure 8 Step S84: Obtain the object length corresponding to associated object 20.

[0163] In step S84, the processor 200 can obtain a preset object length, or filter the obtained current detection length to obtain the object length, or use the distance between the fourth node and the fifth node on the target object 10 as the object length. For details, please refer to [link to relevant documentation]. Figures 3 to 7 The details and related descriptions will not be repeated here.

[0164] Please see Figure 11 In some embodiments, the method for obtaining the object length corresponding to the associated object 20 may include the following steps S111 to S114:

[0165] Step S111: Monitor the current detection length of associated object 20.

[0166] It is understood that in this embodiment, the detection device 500 can continuously collect detection information and output the collected detection information to the processor 200. The processor 200 can monitor the current detection length of the associated object 20 through the continuously acquired detection information.

[0167] Step S112: Determine whether the monitoring results of the current detection length meet the screening criteria.

[0168] It is understood that if the monitoring result for the current detection length does not meet the screening criteria, the process returns to step S111 to continue monitoring the current detection length of the associated object 20. If the monitoring result for the current detection length meets the screening criteria, the process proceeds to step S113.

[0169] In step S112, the processor 200 can determine whether the monitoring results of the current detection length meet the screening criteria by determining whether the monitoring results of the current detection length meet the quantity and / or time conditions. For details, please refer to [link to relevant documentation]. Figures 3 to 7 The details and related descriptions will not be repeated here.

[0170] Step S113: Select the current detection length with the largest value from multiple current detection lengths.

[0171] Step S114: Determine the length of the currently detected object with the associated object 20.

[0172] It can be understood that by filtering object lengths from multiple current detection lengths, the processor 200 can determine the object length only for the associated objects 20 corresponding to the multiple currently detected lengths. When another user moves the corresponding associated object 20 to the detection area, the processor 200 can first collect multiple current detection lengths for the other user's associated object 20, and then determine the object length from among the multiple current detection lengths corresponding to that user's associated object 20. In this way, the robot can recognize and respond to the trigger commands of multiple users.

[0173] Please see Figure 12 In other embodiments, the method for obtaining the object length corresponding to the associated object 20 may include the following steps S121 and S122:

[0174] Step S121: Obtain the distance between the fourth node and the fifth node as the third length;

[0175] In step S121, the processor 200 can obtain the straight-line distance between the fourth node and the fifth node as the third length, as detailed in [reference needed]. Figures 3 to 7 The details and related descriptions will not be repeated here.

[0176] Step S122: Determine the third length as the object length corresponding to the associated object 20.

[0177] It is understandable that when the target object 10 and the associated object 20 are integrated, the size of the associated object 20 can correspond to the size of the target object 10. The processor 200 determines the distance between the fourth node and the fifth node on the target object 10 as the object length, which simplifies the process of the processor 200 obtaining the associated length.

[0178] Please continue reading. Figure 8 Step S85: Calculate the association distance based on the object length corresponding to the associated object 20 and the current detection length.

[0179] In step S85, the processor 200 can calculate the correlation distance according to the Pythagorean theorem, as detailed in the following document. Figures 3 to 7 The details and related descriptions will not be repeated here.

[0180] Step S86: Determine whether the association distance meets the association distance condition.

[0181] It is understandable that, in response to the association distance not meeting the association distance condition, the process returns to step S81 to re-obtain the target distance corresponding to target object 10. In response to the association distance meeting the association distance condition, the process proceeds to step S87.

[0182] In some embodiments, in response to the association distance not meeting the association distance condition, step S83 can be returned to reacquire the current detection length of the associated object 20. Figure 8 The example illustrates the situation where the association distance does not meet the association distance condition and the process returns to step S81.

[0183] In step S86, the processor 200 can determine whether the association distance meets the association distance condition based on the numerical value of the association distance, or it can determine whether the association distance meets the association distance condition based on the numerical value of the result distance after calculating the association distance. For details, please refer to [link to relevant documentation]. Figures 3 to 7 The details and related descriptions will not be repeated here.

[0184] Step S87: Determine that the associated object 20 executes a trigger action.

[0185] It is understood that the processor 200 can calculate the association distance based on the object length corresponding to the associated object 20 and the current detection length, and then determine whether the associated object 20 has executed a trigger action based on the value of the association distance. This avoids the situation where the processor 200 misidentifies the shadow of the associated object 20 on the projection area 401 as the associated object 20 when identifying the associated object 20 in the depth image and determining whether the associated object 20 has executed a trigger action based on the depth value of the identified associated object 20, thus obtaining an incorrect depth value for determining whether the associated object 20 has executed a trigger action. This improves the accuracy of identifying the associated object 20 and the accuracy of determining whether the associated object 20 has executed a trigger action, i.e., it improves the accuracy of action recognition.

[0186] Please see Figure 13 , Figure 13 This is a schematic diagram of a charging method provided in another embodiment of this application. For example... Figure 13 As shown, the action recognition method may include steps S131 to S139. Specifically, steps S131 to S137 are related to... Figure 8 Steps S81 to S87 are the same as or similar to those in the previous steps, and will not be repeated here. Figure 8 compared to, Figure 13 The action recognition method also includes steps S138 and S139.

[0187] Step S138: Determine the type of trigger action.

[0188] It is understood that trigger actions can include click actions and swipe actions. In step S138, the processor 200 can determine the type of trigger action based on the distance between the start position information and end position information corresponding to the associated object 20.

[0189] Please see Figure 14In some embodiments, the method for determining the type of triggering action may include the following steps S141 to S146:

[0190] Step S141: Obtain the starting position information corresponding to the associated object 20.

[0191] In step S141, the processor 200 can establish a detection coordinate system based on the detection information, and determine the position of the preset node of the associated object 20 when the associated object 20 begins to execute the triggering action in the detection coordinate system, that is, determine the starting position information. See details [link to relevant documentation]. Figures 3 to 7 The details and related descriptions will not be repeated here.

[0192] Step S142: Obtain the end position information corresponding to the associated object 20.

[0193] In step S142, the processor 200 can establish a detection coordinate system based on the detection information, and determine the position of the preset node of the associated object 20 when the current detection length is the object length, i.e., determine the end position information. See details in [link to documentation]. Figures 3 to 7 The details and related descriptions will not be repeated here.

[0194] In this embodiment, the execution order of steps S141 and S142 is not limited. For example, step S141 may be executed before step S142. Alternatively, step S141 may be executed after step S142. Yet another example is that steps S141 and S142 may be executed simultaneously, and step S143 will proceed after both steps S141 and S142 have been completed. Figure 14 The example is given that step S141 can be performed before step S142.

[0195] Step S143: Determine the movement distance of the preset node based on the starting position information and the ending position information.

[0196] In step 143, the processor 200 can calculate the movement distance of the preset node based on the coordinate data corresponding to the starting position information and the coordinate data corresponding to the ending position information. See details in [link to documentation]. Figures 3 to 7 The details and related descriptions will not be repeated here.

[0197] Step S144: Determine whether the movement distance meets the movement distance condition.

[0198] It is understandable that if the movement distance does not meet the movement distance condition, the process can proceed to step S145; if the movement distance meets the movement distance condition, the process can proceed to step S146.

[0199] In step S144, the processor 200 can determine whether the movement distance meets the movement distance condition based on the numerical value of the movement distance. (See details below.) Figures 3 to 7 The details and related descriptions will not be repeated here.

[0200] Step S145: Determine that the associated object 20 has performed a click action.

[0201] Step S146: Determine that the associated object 20 has performed a sliding action.

[0202] It can be understood that the processor 200 can determine the specific type of the triggered action by measuring the distance the device moves in a direction other than the direction from the detection position toward the projection area 401, based on the value of the distance moved at one end (e.g., the fingertip) corresponding to the preset node of the associated action.

[0203] It is understandable that by acquiring and comparing the starting position information and the interpretation position information corresponding to the associated object 20, after determining that the associated object 20 has executed a triggering action, the movement distance of the preset node can be further determined, and the type of triggering action can be further determined based on the movement distance of the preset node, so that the processor 200 can respond in a timely manner to the determined type of triggering action, thereby improving the user experience.

[0204] Please continue reading. Figure 13 Step S139: Adjust the image projected in projection area 401 according to the type of the triggered action and the starting position information of the preset node.

[0205] In step S139, the processor 200 can adjust the image of the specified display area 402 in the projection area 401 according to the type of trigger action and the starting position information. (See details in [link to relevant documentation]). Figures 3 to 7 The details and related descriptions will not be repeated here.

[0206] It is understood that the beneficial effects of the action recognition method in the embodiments of this application can be specifically referred to in the robot and its related descriptions, and will not be repeated here.

[0207] Please see Figure 15 , Figure 15 This is a schematic diagram of an action recognition system 600 according to an embodiment of this application.

[0208] The action recognition system 600 provided in this application embodiment may include a target distance acquisition module 601, a target distance judgment module 602, a current detection length acquisition module 603, an object length acquisition module 604, a calculation module 605, and an association distance judgment module 606.

[0209] The target distance acquisition module 601 is used to acquire the distance between the target object 10 and the detection position, that is, to acquire the target distance.

[0210] like Figure 4 As shown in the example, the target distance acquisition module 601 can identify the target object 10 in the depth image and obtain the depth value corresponding to the target object 10 as the target distance.

[0211] The target distance determination module 602 is used to determine whether the target distance meets the target distance conditions.

[0212] The current detection length acquisition module 603 is used to acquire the current detection length corresponding to the associated object 20.

[0213] The object length acquisition module 604 is used to obtain the object length corresponding to the associated object 20.

[0214] It is understandable that the object length acquisition module 604 can acquire a preset object length, or it can filter out the object length from multiple currently detected lengths.

[0215] The calculation module 605 is used to calculate the association distance based on the current detection length and the object length.

[0216] It is understandable that the calculation module 605 can use the Pythagorean theorem to calculate the current detection length and the object length, thereby obtaining the association distance. The association distance is the distance from the detection position toward the projection area 401.

[0217] The association distance judgment module 606 is used to determine whether the association distance meets the association distance condition.

[0218] It is understandable that when the association distance judgment module 606 determines that the association distance meets the association distance condition, it can determine that the associated object 20 will execute a trigger action.

[0219] It is understood that the division of the various modules in the above-described motion recognition system 600 is only for illustrative purposes. In other embodiments, the motion recognition system 600 can be divided into different modules as needed to complete all or part of the functions of the above-described motion recognition system 600.

[0220] The specific implementation of each module in the embodiments of this application can also be referred to accordingly. Figures 8 to 14 The corresponding description of the method embodiments shown.

[0221] It is understood that the beneficial effects of the action recognition system 600 in the embodiments of this application can be specifically referred to in the robot and its related descriptions, and will not be repeated here.

[0222] Based on the same concept, embodiments of this application also provide a storage medium. The readable storage medium stores computer instructions that, when executed on the processor 200, cause the processor 200 to perform the action recognition method provided in the foregoing embodiments.

[0223] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An action recognition method, characterized in that, The action recognition method includes: Obtain the current detection length corresponding to the associated object, where the current detection length is the projection length of the associated object; Get the length of the object corresponding to the associated object; The association distance is calculated based on the length of the object corresponding to the associated object and the current detection length. In response to the association distance satisfying the association distance condition, the associated object is determined to perform a trigger action; The associated object includes a first node, a second node, and a third node. The third node is located between the first node and the second node. The portion between the third node and the second node is capable of bending. Obtaining the current detection length corresponding to the associated object includes: The distance between the first node and the second node is obtained as the first length; The second length is calculated based on the distance between the first node and the third node; The length with the larger value between the first length and the second length is taken as the current detection length.

2. The action recognition method according to claim 1, characterized in that, The action recognition method includes: Obtain the target distance corresponding to the target object, wherein the associated object moves in association with the target object, and the target distance is the distance between the target object and the detection position, and the direction of the associated distance corresponds to the direction of the target distance; In response to the target distance satisfying the target distance condition, the process proceeds to calculate the associated distance based on the object length corresponding to the associated object and the current detection length. If the target distance does not meet the target distance condition, or if the associated distance does not meet the associated distance condition, the target distance corresponding to the target object is returned.

3. The action recognition method according to claim 2, characterized in that, The step of obtaining the target distance corresponding to the target object includes: Acquire depth images; Identify the target object in the depth image; Obtain the depth value of the target object in the depth image; The depth value is determined as the target distance corresponding to the target object.

4. The action recognition method according to claim 1, characterized in that, Obtaining the object length corresponding to the associated object includes: Monitor the current detection length of the associated object; In response to the monitoring result of the current detection length satisfying the screening conditions, the current detection length with the largest value is selected from multiple current detection lengths, and the screening conditions include at least one of quantity conditions and time conditions; The current detection length selected is determined to be the object length of the associated object.

5. The action recognition method according to claim 2, characterized in that, The target object includes a fourth node and a fifth node, and obtaining the object length corresponding to the associated object includes: The distance between the fourth and fifth nodes is the third length. The third length is determined to be the object length corresponding to the associated object.

6. The action recognition method according to claim 1, characterized in that, The action recognition method includes: Obtain the starting position information corresponding to the associated object, wherein the starting position information corresponds to the position of a preset node of the associated object when the current detection length is the length of the object; Obtain the end position information of the associated object, wherein the end position information corresponds to the location of the preset node of the associated object when the association distance meets the association distance condition; The movement distance of the preset node is determined based on the starting position information and the ending position information; Determine whether the movement distance meets the movement distance condition; In response to the movement distance satisfying the movement distance condition, it is determined that the associated object performs a first action; In response to the movement distance not meeting the movement distance condition, it is determined that the associated object will perform a second action.

7. An action recognition system, characterized in that, include: The current detection length acquisition module is used to acquire the current detection length of the associated object, wherein the current detection length is the projection length of the associated object in the direction perpendicular to the projection area; The object length module is used to obtain the object length corresponding to the associated object, wherein the object length is the actual length of the associated object; The calculation module is used to calculate the association distance based on the object length corresponding to the associated object and the current detection length, wherein the association distance is the distance the associated object moves in a direction perpendicular to the projection area; The association distance determination module is used to determine that the associated object performs a trigger action in response to the association distance meeting the association distance condition. The associated object includes a first node, a second node, and a third node. The third node is located between the first node and the second node. The portion between the third node and the second node can be curved. When the current detection length acquisition module acquires the current detection length of the associated object, the current detection length acquisition module is used to acquire the distance between the first node and the second node as a first length, calculate a second length based on the distance between the first node and the third node, and acquire the length with the larger value between the first length and the second length as the current detection length.

8. A robot, characterized in that, include: The robot itself; Memory, used to store computer programs; A processor is configured to execute the computer program stored in the memory, wherein when the computer program is executed, the processor is configured to perform the action recognition method as described in any one of claims 1 to 6.

9. A storage medium, characterized in that, The storage medium includes a computer program that, when run on an electronic device, causes the electronic device to perform the action recognition method as described in any one of claims 1 to 6.

Images