Method, device and human-computer interaction system for determining target object

Abstract

The embodiments of the present application disclose a method, device, and human-computer interaction system for determining a target object. The method of one embodiment of the present application includes: in response to detecting a position change of an object, determining a sequence of image frames to be detected based on the detection moment of the position change; performing human key point detection on the image frames to be detected in the sequence of frames to be detected; and determining a target object that performs a target operation on the object based on the detection result of the human key point detection. This implementation method is conducive to the accuracy of the determined target object.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. patent application No. 16 / 528,134, filed on July 31, 2019, entitled “Method And Apparatus for Determining a Target Object, and Human-Computer Interaction System.” Technical Field

[0003] The embodiments of the present application relate to the field of computers, specifically to the field of image recognition, and in particular to a method, device, and human-computer interaction system for determining a target object. Background Art

[0004] A smart retail store is a new retail experience where customers can freely choose goods in the store and the bill is automatically pushed to the customers at the end of the shopping without having to queue up to check out.

[0005] In order to autonomously provide retail services to customers, it is necessary to automatically determine which customer purchased which product. In other words, it is necessary to identify which product on which shelf was taken by which customer. Summary of the invention

[0006] The embodiments of the present application provide a method, an apparatus, and a human-computer interaction system for determining a target object.

[0007] In a first aspect, an embodiment of the present application provides a method for determining a target object for performing a target operation behavior, including: in response to detecting a position change of an object, determining a sequence of image frames to be detected based on a detection moment when the position change is detected; performing human key point detection on the image frames to be detected in the sequence of frames to be detected; and determining the target object for performing the target operation behavior on the object based on the detection result of the human key point detection.

[0008] In some embodiments, in response to detecting a position change of an object, a sequence of image frames to be detected is determined based on a detection moment when the position change is detected, including: in response to detecting a position change of the object, determining a detection moment when the position change is detected; arranging image frames whose time difference between the image acquisition moment and the detection moment is less than a preset time length in chronological order of the acquisition moments to form a sequence of image frames to be detected.

[0009] In some embodiments, performing human key point detection on the image frames to be detected in the frame sequence to be detected includes: performing human key point detection on the target image frames whose image acquisition time is closest to the detection time in the frame sequence to be detected.

[0010] In some embodiments, in response to detecting a position change of an object, a sequence of image frames to be detected is determined based on a detection moment when the position change is detected, including: in response to detecting a position change of the object, a target image acquisition device for acquiring the sequence of image frames to be detected is determined based on the position of the object; from the sequence of image frames acquired by the target image acquisition device, image frames acquired at an acquisition moment whose time difference with the detection moment is less than a preset time length are determined to form the sequence of image frames to be detected.

[0011] In some embodiments, based on the detection results of human key point detection, the target object for performing the target operation on the object is determined, including: determining the target object based on the distance between each human object and the object in the detection results of human key point detection of the target image frame.

[0012] In some embodiments, the distance between the human object in the target image frame and the object is determined by the following distance determination steps: for the human object in the target image frame, detect whether there is a hand key point indicating the hand of the human object in the target image frame; if so, use the distance between the hand key point of the human object and the object as the distance between the human object and the object.

[0013] In some embodiments, the distance determination step also includes: for a human object in a target image frame, if there is no key point in the target image frame for indicating the hand of the human object, selecting the key point with the highest priority among the human key points of the human object existing in the target image frame according to the key point priority order; and using the distance between the selected human key point and the object as the distance between the human object and the object.

[0014] In some embodiments, based on the detection results of human key point detection, the target object that performs the target operation on the object is determined, including: based on the detection results of human key point detection of each image frame to be detected, generating the movement trajectory of each human object in the image frame sequence to be detected; determining the trajectory point in each movement trajectory that is closest to the object as a candidate trajectory point; and determining the target object based on the distance between the determined candidate trajectory point and the object.

[0015] In some embodiments, for a human object in an image frame to be detected, if the detection result of the human key point detection of the human object indicates that there are hand key points of the human object in the image frame to be detected, the hand key points are used as trajectory points on the trajectory of the human object.

[0016] In some embodiments, for a human object in an image frame to be detected, if the detection result of the human key point detection of the human object indicates that the hand key point of the human object does not exist in the image frame to be detected, then according to the key point priority order, the key point with the highest priority among the human key points of the human object existing in the image frame to be detected is selected as the trajectory point on the trajectory of the human object.

[0017] In some embodiments, in the detection results of the human body key point detection, the position of each human body key point is the position in a preset three-dimensional coordinate system.

[0018] In some embodiments, human key point detection is performed on image frames to be detected in a sequence of frames to be detected, including: for image frames to be detected in the sequence of frames to be detected, determining the object area where each human object is located in the image frames to be detected; performing human key point detection on each object area to obtain a detection result of the image frames to be detected.

[0019] In a second aspect, an embodiment of the present application further provides a human-computer interaction system, including: a shelf on which items are arranged, at least one shelf separating preset areas to form a channel; an image acquisition device, acquiring images within at least one channel; a sensor, detecting whether an item arranged on the shelf changes position; a processor, in response to detecting a position change of the item, determining a sequence of image frames to be detected based on a detection moment when the position change is detected; performing human body key point detection on the image frames to be detected in the sequence of frames to be detected; and determining a target object for performing a target operation on the item based on the detection result of the human body key point detection.

[0020] In some embodiments, the image acquisition device is a depth camera.

[0021] In the third aspect, an embodiment of the present application provides a device for determining a target object for performing a target operation behavior, including: a unit for determining a sequence of image frames to be detected, configured to determine the sequence of image frames to be detected based on a detection moment when the position change is detected in response to detecting a position change of an object; a human key point detection unit, configured to perform human key point detection on the image frames to be detected in the sequence of frames to be detected; and a target object determination unit, configured to determine the target object for performing the target operation behavior on the object based on the detection result of the human key point detection.

[0022] In some embodiments, the unit for determining the sequence of image frames to be detected is further configured to: determine the detection moment when the position change is detected in response to detecting a position change of an object; and arrange the image frames whose time difference between the image acquisition moment and the detection moment is less than a preset time length in chronological order of the acquisition moments to form a sequence of image frames to be detected.

[0023] In some embodiments, the human key point detection unit is further configured to: perform human key point detection on a target image frame whose image acquisition time is closest to the detection time in the frame sequence to be detected.

[0024] In some embodiments, the unit for determining the sequence of image frames to be detected is further configured to: in response to detecting a position change of an object, determine a target image acquisition device for acquiring the sequence of image frames to be detected based on the position of the object; and from the sequence of image frames acquired by the target image acquisition device, determine that the image frames acquired at the acquisition moment whose time difference with the detection moment is less than a preset time length form the sequence of image frames to be detected.

[0025] In some embodiments, the target object determination unit is further configured to: determine the target object based on the distance between each human object and the object in the detection result of the human key point detection of the target image frame.

[0026] In some embodiments, the target object determination unit determines the distance between the human object and the object in the target image frame through the following distance determination steps: for the human object in the target image frame, detect whether there is a hand key point indicating the hand of the human object in the target image frame; if so, use the distance between the hand key point of the human object and the object as the distance between the human object and the object.

[0027] In some embodiments, the distance determination step also includes: for a human object in a target image frame, if there is no key point in the target image frame for indicating the hand of the human object, selecting the key point with the highest priority among the human key points of the human object existing in the target image frame according to the key point priority order; and using the distance between the selected human key point and the object as the distance between the human object and the object.

[0028] In some embodiments, the target object determination unit is further configured to: generate a movement trajectory of each human object in the sequence of image frames to be detected based on the detection results of the human key point detection of each image frame to be detected; determine the trajectory point in each movement trajectory that is closest to the object as a candidate trajectory point; and determine the target object based on the distance between the determined candidate trajectory point and the object.

[0029] In some embodiments, for a human object in an image frame to be detected, if the detection result of the human key point detection of the human object indicates that there are hand key points of the human object in the image frame to be detected, the hand key points are used as trajectory points on the trajectory of the human object.

[0030] In some embodiments, for a human object in an image frame to be detected, if the detection result of the human key point detection of the human object indicates that the hand key point of the human object does not exist in the image frame to be detected, then according to the key point priority order, the key point with the highest priority among the human key points of the human object existing in the image frame to be detected is selected as the trajectory point on the trajectory of the human object.

[0031] In some embodiments, in the detection results of the human body key point detection, the position of each human body key point is the position in a preset three-dimensional coordinate system.

[0032] In some embodiments, the human key point detection unit can be further configured to: for the image frame to be detected in the frame sequence to be detected, determine the object area where each human object is located in the image frame to be detected; perform human key point detection on each object area to obtain the detection result of the image frame to be detected.

[0033] In a fourth aspect, an embodiment of the present application provides an electronic device, comprising: an interface; one or more processors; a memory on which one or more programs are stored, and one or more processors are operatively connected to the above-mentioned interface and the above-mentioned memory, and are used to: in response to detecting a position change of an object, determine a sequence of image frames to be detected based on the detection moment of detecting the position change; perform human body key point detection on the image frames to be detected in the sequence of frames to be detected; and determine a target object for performing a target operation on the object based on the detection results of the human body key point detection.

[0034] In the fifth aspect, an embodiment of the present application also provides an electronic device, one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the method for determining a target object for performing a target operation behavior as described in the first aspect.

[0035] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, wherein when the above-mentioned computer program is executed by one or more processors, the above-mentioned one or more processors: in response to detecting a position change of an object, determine a sequence of image frames to be detected based on the detection moment of detecting the position change; perform human body key point detection on the image frames to be detected in the sequence of frames to be detected; and determine a target object for performing a target operation on the object based on the detection result of the human body key point detection.

[0036] In the seventh aspect, an embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon, wherein when the above-mentioned computer program is executed by one or more processors, the above-mentioned one or more processors execute the method for determining the target object for performing the target operation behavior described in the above-mentioned first aspect.

[0037] The solution for determining the target object for performing the target operation behavior provided by the embodiment of the present application first responds to the detection of the position change of the object, determines the image frame sequence to be detected based on the detection time of the detected position change, then performs human key point detection on the image frame to be detected in the frame sequence to be detected, and finally, based on the detection result of the human key point detection, determines the target object for performing the target operation behavior on the object in the image frame to be detected based on the result of the key point detection performed on the human object in the image frame to be detected, thereby achieving the goal of determining the target object for performing the target operation behavior on the object in the image frame to be detected based on the result of the key point detection performed on the human object in the image frame to be detected when it is determined that the position change of the object occurs, which is beneficial to the accuracy of the determined target object. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0039] Figure 1 is an exemplary system architecture diagram to which a method for determining a target object for performing a target operation behavior according to an embodiment of the present application can be applied;

[0040] Figure 2 is a flow chart of an embodiment of a method for determining a target object for performing a target operation behavior according to the present application;

[0041] Figure 3 It is a schematic region division method of an image frame to be detected in the method for determining a target object for performing a target operation behavior according to the present application;

[0042] Figure 4 is a schematic diagram of an application scenario of a method for determining a target object for performing a target operation behavior according to the present application;

[0043] Figure 5 is a flow chart of another embodiment of a method for determining a target object for performing a target operation behavior according to the present application;

[0044] Figure 6 is a schematic diagram of an application scenario of the method for determining a target object for performing a target operation behavior of the present application;

[0045] Figure 7It is a schematic flow of determining the distance between each human object and an object in a target image frame in some optional implementations of the method for determining a target object for performing a target operation behavior according to the present application;

[0046] Figure 8 is a structural diagram of an embodiment of an apparatus for determining a target object for performing a target operation behavior according to the present application;

[0047] Fig. 9 It is a structural diagram of a computer system of an electronic device suitable for implementing the method for determining a target object for performing a target operation behavior according to an embodiment of the present application. DETAILED DESCRIPTION

[0048] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It should also be noted that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0049] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0050] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0051] Figure 1 An exemplary system architecture 100 is shown to which an embodiment of a method for determining a target object for performing a target operation behavior or an apparatus for determining a target object for performing a target operation behavior of the present application can be applied.

[0052] like Figure 1 As shown, the system architecture 100 may include image acquisition devices 101, 102, 103, a network 104, and a server 105. The network 104 is used to provide a medium for communication links between the image acquisition devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0053] The image acquisition devices 101, 102, 103 can transmit the acquired images to the server 105 via the network 104. The image acquisition devices 101, 102, 103 can be various devices with image acquisition functions, including but not limited to separately arranged cameras or image acquisition units integrated in other electronic devices, such as but not limited to smart phones, tablet computers, laptop computers, desktop computers, etc.

[0054] The server 105 may be a server that provides various services, such as a background server that processes the images captured by the image acquisition devices 101, 102, and 103. The background server may perform data processing based on the received images, and send payment prompt information to an associated terminal (not shown) based on the processing results (e.g., information indicating a person object that picks up a commodity).

[0055] It should be noted that the method for determining the target object for performing the target operation behavior provided in the embodiment of the present application can be executed by the server 105. Accordingly, the device for determining the target object for performing the target operation behavior can be set in the server 105. Alternatively, the method for determining the target object for performing the target operation behavior provided in the embodiment of the present application can be executed by a processor integrated in the image acquisition device 101, 102, 103. Accordingly, the device for determining the target object for performing the target operation behavior can be set in the processor on the image acquisition device 101, 102, 103.

[0056] It should be understood that Figure 1 The number of image acquisition devices, networks and servers in the embodiment is only for illustration. Any number of image acquisition devices, networks and servers may be provided as required. For example, the server may be a clustered server, including multiple servers with different processes deployed.

[0057] Continue to refer Figure 2 , which shows a process 200 of an embodiment of a method for determining a target object for performing a target operation behavior according to the present application.

[0058] The method for determining a target object for performing a target operation behavior comprises the following steps:

[0059] Step 201 : in response to detecting a position change of an object, a sequence of image frames to be detected is determined based on a detection time when the position change is detected.

[0060] Here, an item can be any physical goods, for example, a commodity placed on a shelf for customers to select and purchase.

[0061] The position change of an item can be detected in various feasible ways. For example, in some optional implementations, a plurality of gravity sensors can be provided on a shelf where items are placed, and if one or several gravity sensors on the shelf detect a gravity change, it can be determined that the position of the item placed on the shelf has changed.

[0062] Alternatively, in some other optional implementations, multiple image acquisition devices may be provided, each of which may acquire images of the area of ​​the shelf where the items are placed at a certain sampling interval. If there are differences between the images acquired at adjacent sampling moments, it may be determined that there is a change in the position of the items within the acquisition range of the image acquisition device.

[0063] Alternatively, in some other optional implementations, the shelf where the items are placed may also be provided with a position sensor (for example, including but not limited to a photoelectric position sensor, a magnetic induction position sensor, etc.). If one or more position sensors detect a position change, it can be determined that the position of the items placed on the shelf has changed.

[0064] In addition, it can be understood that in some optional implementations, a combination of multiple sensors can also be used to detect the position change of an object. For example, the probability values ​​of the existence of the position change of the object determined based on any at least two of the sensors such as the gravity sensor, the image acquisition device, the position sensor, etc. can be weighted, and whether the position change of the object exists can be determined based on the weighted sum.

[0065] In the case where it is determined based on the detection result of the sensor that there is a change in the position of the object, the execution entity may further determine the time when the change in the position of the object occurs.

[0066] In some optional implementations, the moment when the execution subject makes the judgment result that "the position of the object has changed" can be directly used as the detection moment.

[0067] Alternatively, in some other optional implementations, when the execution subject makes the judgment result that "the position of the object has changed", the acquisition time of the judgment basis for making this judgment result is used as the detection time when the position change is detected. Specifically, for example, in some application scenarios, the images collected by the image acquisition device at adjacent sampling times are used to determine whether there is a position change of the object. Assuming that the execution subject receives an image collected at time t1 and an image collected at time t2 (t2>t1), and determines that the image collected at time t2 is different from the image collected at time t1 (that is, the position of the object has changed), then time t2 can be used as the detection time when the position change is detected.

[0068] According to the determined detection time when the position change is detected, the image frame sequence to be detected can be determined.

[0069] Here, the sequence of image frames to be detected can be a sequence formed by image frames captured by an image acquisition device arranged near the object whose position has changed and used to capture images of an object (e.g., a person) performing a target operation (e.g., picking up) on the object, arranged in chronological order of acquisition time.

[0070] In some optional implementations, if a position change of an object is detected, the detection time can be determined in the above manner. Then, the image frames captured by the image acquisition device, where the time difference between the image acquisition time and the detection time is less than a preset time length, can be arranged in the order of the acquisition time to form a sequence of image frames to be detected.

[0071] For example, in some application scenarios of these optional implementations, if the detection time of the position change is 10:01:58 am on March 28, 2019, then the image frames to be detected acquired between 10:00:58 am on March 28, 2019 and 10:02:58 am on March 28, 2019 can be arranged in chronological order of acquisition time to form an image frame sequence as the image frame sequence to be detected.

[0072] Step 202: perform human body key point detection on the image frames to be detected in the sequence of frames to be detected.

[0073] Human key point detection generally refers to determining one or more key points belonging to a certain human body from an image. Human key points may be, for example, points that can reflect key information of a certain aspect of the human body. Specifically, in the method for determining the target object that performs the target operation behavior in each embodiment of the present application, in order to determine the object that performs the target operation behavior (i.e., the target object), the human key points may be, for example, feature points that can reflect the posture of the human object in the image frame to be detected. For example, the human key points may include points for characterizing the head, neck, shoulders, elbows, hands, arms and other parts of the human body. Through human key point detection, the positions of the various key points of the human object in the image frame to be detected can be determined, and the posture of the human object in the image frame to be detected can be determined based on the positions of the various key points.

[0074] In some optional implementations, when performing human key point detection on an image frame to be detected in a sequence of frames to be detected, the object region where the human object is located in the image frame to be detected can be first determined, and then human key point detection can be performed in the determined object region to obtain a detection result of the image frame to be detected. In this way, by first determining the region where the human object is located and then performing human key point detection in the region, it is ensured that the human key points detected are in the region where the human object is located, thereby improving the reliability of human key point detection.

[0075] It is understandable that if the image frame to be detected contains multiple human objects, the human key point detection can be performed on the multiple human objects contained in the image frame to be detected, so as to obtain the detection result. For example, the object area where each human object in the image frame to be detected is located can be first determined, and then the human key point detection can be performed on the determined object area, so as to obtain the human key point of each human object in the image frame to be detected.

[0076] Here, any feasible algorithm can be used to detect key points of the human body, for example, a top-down detection method can be used, or a bottom-up detection method can be used.

[0077] Among them, the top-up human skeleton key point positioning algorithm mainly includes two parts, human body detection and single human body key point detection, that is, firstly, each person is detected by the target detection algorithm, and then the human skeleton key point detection is performed for a single person based on the detection frame. For example, CPM (Convolutional Pose Machines) can be used for key point detection. Specifically, the network structure is divided into multiple stages, wherein the first stage will produce a preliminary key point detection effect, and the following stages are all input with the predicted output of the previous stage and the features extracted from the original image, further improving the key point detection effect. Alternatively, CPN (Cascaded Pyramid Network) can also be used for key point detection. Specifically, CPN includes GlobalNet (global network) and RefineNet (refinement network), GlobalNet is used to detect key points that are easy to detect and difficult to detect, and RefineNet is used to detect more difficult or invisible key points, and the union of the two detection results can be used as the final detection result of human key point detection.

[0078] The bottom-up method also includes two parts, key point detection and key point clustering, that is, first all the key points in the image need to be detected, and then all the key points are clustered into different individuals through relevant strategies. In the bottom-up method, how to accurately connect and cluster the detected key points is the key to the accuracy of the final detection result. For example, the Part Segmentation algorithm can be used to cluster key points. Specifically, the human body is segmented into different parts, and the key points fall at specific positions in the segmented area. The relationship between the key points is modeled through part segmentation, which can not only explicitly provide spatial prior knowledge of the key points of the human body to guide the learning of the network, but also play a corresponding role in connecting the key points when clustering different key points of the human body in the end. Alternatively, the PAFs (Part Affinity Fields) algorithm can be used for modeling. Specifically, by modeling different limb structures of the human body and using vector fields to simulate different limb structures, the problem of misconnection caused by simply using the middle point on the limb can be solved.

[0079] The detection result obtained through step 202 may include, for example, identification information of key points of the human body detected and corresponding position information.

[0080] Here, the identification information may be, for example, any information that can characterize the identity of a key point of a human body, such as information used to characterize the human object to which the key point of a human body belongs (e.g., human object A or human object B), and information about the type of the key point (e.g., the key point characterizes the head, elbow, hand, or other part of the human object).

[0081] In addition, the position information may be any information that can qualitatively or quantitatively describe the actual position of the key point of the human body. For example, the position information may be the coordinates of the key point of the human body in a pre-established coordinate system.

[0082] Alternatively, the position information may also be descriptive information that qualitatively describes the actual position of the key points of the human body. For example, the image frame to be detected may be divided into three regions of left, middle and right in the horizontal direction, and the image frame to be detected may be divided into three regions of upper, middle and lower in the vertical direction. Then, Figure 3 As shown, the image frame 300 to be detected can be divided into 9 areas (i.e., upper left, upper middle, upper right, middle left, middle middle, middle right, lower left, lower middle and lower right). Then, the position information of the key point of the human body can be information for indicating which area of ​​the 9 areas the key point of the human body is located in.

[0083] Alternatively, the position information can also be a qualitative description of whether the key point of the human body is in a specific area of ​​the image frame to be detected. For example, the position information can be used to describe whether the key point of the human body is in a specific area according to Figure 3 The “upper right” area of ​​the image frame to be detected is divided in the manner shown.

[0084] Step 203: Based on the detection result of the key point detection of the human body, determine the target object that performs the target operation behavior on the object.

[0085] As described in step 202, through human key point detection, the identification information and corresponding position information of the human key points in the image frame to be detected can be obtained. Accordingly, it can be determined which human object in the image frame to be detected has performed the target operation on the object based on the position information and identification information of the human key points.

[0086] For example, in some application scenarios, the target operation behavior of "picking up an item from a shelf" is detected, that is, it is expected to determine the human object that has performed the target operation behavior of "picking up an item from a shelf" from the image frame to be detected. Then, in these application scenarios, the hand key points of each human object can be determined from the image frame to be detected, and the human object that picks up the item from the shelf can be determined based on the position of the hand key points.

[0087] The solution for determining the target object for performing the target operation behavior provided by the embodiment of the present application first responds to the detection of the position change of the object, determines the image frame sequence to be detected based on the detection time of the detected position change, then performs human key point detection on the image frame to be detected in the frame sequence to be detected, and finally, based on the detection result of the human key point detection, determines the target object for performing the target operation behavior on the object in the image frame to be detected based on the result of the key point detection performed on the human object in the image frame to be detected, thereby achieving the goal of determining the target object for performing the target operation behavior on the object in the image frame to be detected based on the result of the key point detection performed on the human object in the image frame to be detected when it is determined that the position change of the object occurs, which is beneficial to the accuracy of the determined target object.

[0088] Continue to see Figure 4 , Figure 4 It is a schematic diagram 400 of an application scenario of the method for determining a target object for performing a target operation behavior according to this embodiment.

[0089] exist Figure 4 In the application scenario shown, the goal is to identify the person who picks up the product from the shelf.

[0090] In this application scenario, a camera (not shown in the figure) arranged above the shelf 401 can capture images at a certain sampling interval, and transmit the captured images to the processor for caching or storage. In addition, a sensor capable of detecting whether the position of a commodity has changed can be arranged on the shelf. If a position change of a commodity on the shelf is detected, the sensor can send a corresponding trigger signal to the processor, so that the processor can determine the image frame sequence to be detected from the unit that caches or stores the images captured by the camera according to the time when the sensor trigger signal is received.

[0091] Next, the processor may further perform human body key point detection on the image frames to be detected in the sequence of image frames to be detected, thereby obtaining a detection result, which may include, for example, identification information and position information of the human body key points.

[0092] Specifically, in Figure 4 In the application scenario shown, the identification information of the human body key point can be used to indicate the human body object 402 to which the human body key point belongs, and the type of the human body key point. For example, the identification information of the human body key point a may include information indicating the "hand of the object 402", the identification information of the human body key point b may include information indicating the "elbow of the object 402", the identification information of the human body key point c may include information indicating the "shoulder of the object 402", and the identification information of the human body key point e may include information indicating the "head of the object 402".

[0093] In addition, Figure 4 In the application scenario shown, the position information of the key point of the human body may be, for example, information for indicating whether the position of the key point of the human body is within the shelf area. Figure 4 As shown in the dotted frame 403 in FIG. It is understandable that the position and range of the shelf area in the image frame to be detected can be predetermined based on factors such as the relative position between the camera and the shelf 401, the image acquisition range and angle of the camera, and the like.

[0094] If the detection result of the human key point detection indicates that the hands and elbows of the human object 402 are both in the shelf area 403, it can be considered that the human object 402 has performed the action of "picking up the goods from the shelf" on the goods on the shelf.

[0095] It is understandable that the judgment conditions for determining whether a human object has performed the target operation behavior of "picking up a commodity from the shelf" and which human object in the image frame to be detected has performed the target operation behavior of "picking up a commodity from the shelf" can be adjusted according to the application scenario and / or the different presentation forms of the detection results of the human key point detection. For example, if the position information in the detection results of the human key point detection is expressed as the coordinate value of the human key point in the image frame to be detected, the human object that has performed the target operation behavior of "picking up a commodity from the shelf" can be determined based on the distance between the position of the human key point representing the hand and the position of the shelf or the commodity on the shelf.

[0096] Further references Figure 5 , which shows a process 500 of another embodiment of a method for determining a target object for performing a target operation behavior.

[0097] The process 500 of the method for determining a target object for performing a target operation behavior comprises the following steps:

[0098] Step 501 : in response to detecting a position change of an object, determining a target image acquisition device for acquiring a sequence of image frames to be detected based on the position of the object.

[0099] In this step, the method for detecting the change in the position of the object can be as follows: Figure 2 The embodiment shown is performed in a similar manner as step 201. That is to say, for example, a gravity sensor disposed on the shelf can be used to detect gravity changes, and the gravity changes can be used as a basis for determining changes in the position of items. Alternatively, an image acquisition device can be used to capture images of the shelf at a certain sampling interval, and by comparing the two captured images, it is determined whether a change in the position of the item has occurred. Alternatively, a position sensor can be used to directly detect whether a change in the position of the item has occurred. Alternatively, the probability values ​​of the existence of a change in the position of the item determined by any at least two of the above sensors can be weighted, and based on the weighted sum, it can be determined whether a change in the position of the item has occurred.

[0100] Different from step 201, in this step, the target image acquisition device for acquiring the image frame sequence to be detected can be further determined according to the position of the object whose position has changed.

[0101] For example, in some optional implementations, a sensor for detecting a position change of an object may have a position identifier for indicating its position. In these optional implementations, if a sensor detects that a position change of an object has occurred, the execution subject may determine the position where the position change of the object has occurred based on the position identifier of the sensor that detected the position change of the object.

[0102] Furthermore, in these optional implementations, the association relationship between the sensor for detecting the change in the position of the object and the image acquisition device for acquiring the image frames to be detected can be pre-set according to the position of the sensor for detecting the change in the position of the object and the image acquisition range of the image acquisition device for acquiring the image frames to be detected.

[0103] In some application scenarios of these optional implementations, for example, the method for determining the target object performing the target operation behavior of this embodiment is applied to a smart retail store, and is used to determine the human object that picks up the goods on the shelf from the image. The image acquisition device for acquiring the image frame to be detected can be used to acquire images of a certain shelf or a certain area in the smart retail store. In these application scenarios, the shelf area within the image acquisition range of the image acquisition device can be associated with the image acquisition device in advance.

[0104] For example, Figure 6 In the smart retail store 600 shown, there are three rows of shelves (i.e., shelves A to C), and in front of each row of shelves, there is a channel for users to take goods, that is, there are four channels, namely, channel 1 to channel 4, in the smart retail store 600. A plurality of sensors for detecting changes in the position of items (not shown in the figure) can be set in the shelf. An image acquisition device for acquiring images of the channel and the corresponding shelf is set above the channel (e.g., on the ceiling above the channel and / or on the top of the shelf). Figure 6 Schematically shows that 4 groups of image acquisition devices are set in each channel, respectively acquiring images in a certain shelf area in the channel. For example, image acquisition device Aa1 acquires the image of shelf A in area a of channel 1. Figure 6 In the smart retail store 600 shown, the image acquisition device Aa1 can be pre-associated with the shelf area of ​​area a of shelf A in aisle 1. In this way, when a sensor disposed in the shelf area of ​​area a of shelf A in aisle 1 (i.e., a sensor having a position identifier indicating the shelf area of ​​"area a of aisle 1 of shelf A") detects a change in the position of an item, the image acquisition device Aa1 can be determined as a target image sensor.

[0105] It can be understood that the image acquisition device and the shelf area pre-associated therewith can be in a one-to-one correspondence; or, the image acquisition device and the shelf area pre-associated therewith can have a "one-to-many" or "many-to-many" correspondence.

[0106] Specifically, in some application scenarios, each image acquisition device can acquire images in a certain shelf area, and different image acquisition devices are used to acquire images in different shelf areas. Figure 6Taking the smart retail store 600 shown as an example, the image acquisition device Aa1 only acquires images of shelf A in area a of channel 1.

[0107] Alternatively, in other application scenarios, multiple image acquisition devices may be provided to acquire images within a certain shelf area. Figure 6 Taking the smart retail store 600 as an example, a plurality of image acquisition devices for acquiring images in the shelf area may be provided in the shelf area of ​​shelf A in area a of channel 1.

[0108] Alternatively, in other application scenarios, a certain image acquisition device can acquire images in multiple shelf areas. Figure 6 Taking the smart retail store 600 shown as an example, the image acquisition device Aa1 can capture images of shelf A in area a of channel 1, and can also capture images of shelf A in area b of channel 1; similarly, the image acquisition device Ab1 can capture images of shelf A in area b of channel 1, and can also capture images of shelf A in area c of channel 1, and so on.

[0109] It can be understood that if the image acquisition device and the shelf area pre-associated therewith have a "many-to-many" correspondence relationship, there may be multiple target image acquisition devices obtained through step 501.

[0110] Specifically, Figure 6 Take the smart retail store 600 shown in the figure as an example. If in the smart retail store 600, the image acquisition device Aa1 acquires images of the shelf area of ​​shelf A in area a of channel 1 and the shelf area of ​​shelf A in area b of channel 1, and the image acquisition device Ab1 acquires images of the shelf area of ​​shelf A in area b of channel 1 and the shelf area of ​​shelf A in area c of channel 1, then if the sensor set in the shelf area of ​​area b of channel 1 of shelf A (i.e., the sensor with a position mark indicating the shelf area of ​​"area b of channel 1 of shelf A") detects a change in the position of the item, since both the image acquisition device Aa1 and the image acquisition device Ab1 can acquire images of the shelf area of ​​"area b of channel 1 of shelf A", the shelf area can be associated with both the image acquisition device Aa1 and the image acquisition device Ab1. In this way, through this step 501, both the image acquisition device Aa1 and the image acquisition device Ab1 can be determined as the target image acquisition device.

[0111] Step 502: From the image frame sequence captured by the target image capture device, determine the image frames captured at the capture time whose time difference from the detection time is less than a preset time length to form the image frame sequence to be detected.

[0112] In this step, the detection time can be determined by Figure 2 In addition, according to the determined detection moment, the method of determining the image frame captured at the capture moment whose time difference with the detection moment is less than the preset time length from the image frame sequence captured by the target image capture device can also be adopted as in Figure 2 The method described in step 201 of the illustrated embodiment will not be repeated here.

[0113] Different from step 201, if the image acquisition device and the shelf area pre-associated therewith are in a "many-to-many" correspondence relationship, the image frame sequence to be detected obtained by step 502 may be one or more.

[0114] Specifically, as mentioned in the description of step 501, the image acquisition device and the shelf area pre-associated therewith have a "many-to-many" correspondence relationship, which can be understood as follows: a certain image acquisition device can acquire images in multiple shelf areas, and an image in a certain shelf area can be acquired by multiple image acquisition devices. For example, Figure 6 In the smart retail store 600 shown, the image acquisition device Aa1 acquires images of shelf A in area a of channel 1, and also acquires images of shelf A in area b of channel 1. The image acquisition device Ab1 acquires images of shelf A in area b of channel 1, and also acquires images of shelf A in area c of channel 1. In this way, images of shelf A in area b of channel 1 can be acquired by the image acquisition device Aa1 and the image acquisition device Ab1.

[0115] In the application scenario of this "many-to-many" correspondence, there may be multiple target image acquisition devices determined by the above step 501. For example, in the above example, if it is detected that there is a change in the position of items in the shelf area of ​​shelf A in area b of channel 1, the target image acquisition devices corresponding to the shelf area may be image acquisition device Aa1 and image acquisition device Ab1.

[0116] In this way, according to the determined detection moment, a series of image frames captured at the capture moment whose time difference with the detection moment is less than the preset time length can be determined from the image frame sequence captured by the target image capture device Aa1. According to the determined detection moment, a series of image frames captured at the capture moment whose time difference with the detection moment is less than the preset time length can be determined from the image frame sequence captured by the target image capture device Ab1.

[0117] Furthermore, in some application scenarios, a series of image frames captured at a capture moment determined in the target image acquisition device Aa1 and having a time difference with the detection moment less than a preset time length can be taken as a sequence of image frames to be detected (hereinafter referred to as image frame sequence a to be detected), and a series of image frames captured at a capture moment determined in the target image acquisition device Ab1 and having a time difference with the detection moment less than a preset time length can be taken as another sequence of image frames to be detected (hereinafter referred to as image frame sequence b to be detected). That is, in this application scenario, two image frame sequences to be detected can be obtained through step 502, namely, image frame sequence a to be detected and image frame sequence b to be detected.

[0118] Alternatively, in other application scenarios, the image frame sequence a to be detected and the image frame sequence b to be detected acquired by the target image acquisition device Aa1 and the target image acquisition device Ab1 respectively can be spliced ​​in some way, and the spliced ​​image frame sequence can be used as the image frame sequence to be detected obtained by step 502. In these application scenarios, any feasible method can be used for splicing. For example, the image frames can be arranged in the order of the acquisition time of each image frame in the image frame sequence a to be detected and the image frame sequence b to be detected, and the arranged image frame sequence can be used as the image frame sequence to be detected. Alternatively, the image frame sequence b to be detected can be directly spliced ​​to the end of the image frame sequence a to be detected to form the image frame sequence to be detected, or the image frame sequence a to be detected can be directly spliced ​​to the end of the image frame sequence b to be detected to form the image frame sequence to be detected.

[0119] Step 503: perform human body key point detection on the image frames to be detected in the sequence of frames to be detected.

[0120] Step 504: Based on the detection results of the key points of the human body, determine the target object that performs the target operation behavior on the object.

[0121] The above steps 503 and 504 can be respectively adopted Figure 2 Step 202 and step 203 in the illustrated embodiment are performed in a similar manner and will not be described in detail herein.

[0122] The method for determining the target object for performing the target operation behavior of this embodiment is similar to Figure 2Compared with the embodiment shown in the figure, the target image acquisition device can be further determined from the image acquisition devices according to the association relationship between the sensor for detecting the position change of the object and the image acquisition device for acquiring the image. In this way, in an application scenario including multiple image acquisition devices, the image acquisition device that can acquire the image of the area where the position change of the object occurs can be determined according to the association relationship between the image acquisition device and the sensor and the position where the position change of the object is detected, thereby increasing the possibility that each image frame in the sequence of image frames to be detected captures the operation object that causes the position change of the object.

[0123] It can be understood that in some optional implementations of the method for determining the target object for performing the target operation behavior in various embodiments of the present application, the step of performing human key point detection on the image frame to be detected in the sequence of frames to be detected (for example, Figure 2 Step 202 in the illustrated embodiment or Figure 5 Step 503) in the illustrated embodiment can be performed in the following manner:

[0124] Human body key point detection is performed on the target image frame whose image acquisition time is closest to the detection time in the frame sequence to be detected.

[0125] Thus, in these optional implementations, a target image frame can be determined from the sequence of image frames to be detected for key point detection. Thus, on the one hand, the number of image frames that need to be detected for key points can be reduced, and on the other hand, since the target image frame is acquired by the target image acquisition device at the moment closest to the detection moment, the image frame can more objectively reflect the scene at the moment when the object position changes, which is conducive to determining the accuracy of the target object that performs the target operation behavior on the object.

[0126] It is understandable that if the above optional implementation is applied to Figure 5 In step 503 of the illustrated embodiment, the target image frame is an image frame in an image frame sequence acquired by a target image acquisition device.

[0127] Furthermore, in some application scenarios of these optional implementations, if a target image frame is determined from a sequence of image frames to be detected, the detection results based on human key point detection in each embodiment of the present application can be used to determine the target object that performs the target operation on the object, which can be performed in the following manner:

[0128] The target object is determined based on the distance between each human object and the object in the detection result of the human key point detection of the target image frame. For example, the human object closest to the object in the target image frame can be used as the target object. Alternatively, the probability of each human object being the target object can be determined based on the distance between each human object and the object, and the human object with the greatest probability is used as the target object.

[0129] For example, in these application scenarios, the human object that picks up the object (the distance between the human object and the object is 0) in the target image frame can be regarded as the target object that performs the target operation on the object.

[0130] Alternatively, in these application scenarios, although a human object picking up an object is not shown in the target image frame, there is a human object in the target image frame whose distance to the object is smaller than the distance between any other human object in the target image frame and the object. In this example, the human object with the smallest distance to the object can be determined as the target object that performed the target operation (e.g., a "pick up" operation) on the object.

[0131] Here, when determining the distance between the human object and the object, a certain point can be determined from the human object based on the detection results of the human key point detection, and the distance between the point and the object is used as the distance between the human object and the object.

[0132] For example, as an example, assume that two human objects A and B are presented in the target image frame. And in the key point detection result of the target image frame, key points a1 to a7 of human object A and key points b1 to b5 of human object B are detected. In this example, the positions of key points a1 to a7 can be averaged to obtain average position a, and the positions of key points b1 to b5 can be averaged to obtain average position b, and the average position a and average position b are used to represent the positions of human objects A and B in the target image frame, respectively. In this way, the distance d1 between a and the object can be used as the distance between human object A and the object, and the distance d2 between b and the object can be used as the distance between human object B and the object. By comparing d1 and d2, it can be determined which of human object A and human object B is the target object.

[0133] Or, as another example, Figure 7 The process shown is used to determine the distance between each human object and the object in the target image frame.

[0134] Specifically, first, as shown in step 701, a human object (assuming human object A is selected) can be selected from the target image frame, and it is determined whether there is a hand key point indicating the hand of human object A in the target image frame.

[0135] Next, if there is a hand key point of a human object in the human key point detection result of the target image frame, the distance between the hand key point of the human object and the object is used as the distance between the human object and the object, as shown in step 702. For example, if a3 is a hand key point representing the hand of human object A among the key points a1 to a7 of human object A, then the distance between a3 and the object can be used as the distance between human object A and the object.

[0136] Accordingly, if the hand key points of a certain human object do not exist in the human key point detection result of the target image frame, as shown in step 703, the human key points of the human object are selected according to the key point priority order.

[0137] Here, the key point priority order can be pre-set. For example, the key point priority order can be positively correlated with the correlation with the target operation behavior, that is, the higher the correlation with the target operation behavior, the higher the priority of the key point. Specifically, if the target operation behavior is the behavior of "picking up an item", then, obviously, the hand key point has the highest correlation with the behavior of "picking up an item", so in step 701, it is first determined whether there is a hand key point in the target image frame. And because the correlation between the elbow key point, shoulder key point, neck key point, and head key point and the behavior of "picking up an item" gradually decreases, accordingly, when determining the target object of the behavior of "picking up an item", the priority of the key point can be gradually reduced in the order of hand key point→elbow key point→neck key point→head key point, etc.

[0138] Alternatively, the key point priority order may also be related to the posture of the human object in the target image frame. For example, the posture of the human object in the target image frame may be identified, and the key point priority order corresponding to the posture of the human object may be determined based on the correspondence between the preset posture and the key point priority order.

[0139] Next, if the human key points with the highest priority among the human key points of a human object existing in the target image frame are selected through step 703, then in step 704, the distance between the selected human key point and the object can be used as the distance between the human object and the object.

[0140] For example, in the human key point detection result of the target image frame, the key points of the human object B detected are b1 to b5, which are elbow key point b1, neck key point b2, head key point b3, hip key point b4 and leg key point b5. Then, according to the predetermined priority order, it can be known that the elbow key point b1 has the highest priority, so the distance between the elbow key point b1 and the object can be used as the distance between the human object and the object.

[0141] It can be understood that, in this example, if the key point detection result of the human object A in the target image frame includes two key points for indicating the hands of the human object A, which are set as key point a3 and key point a5, then the smaller of the distance between a3 and the object and the distance between a5 and the object can be used as the distance between the human object A and the object.

[0142] In some optional implementations of the embodiments of the present application, based on the detection results of the human body key point detection, a step of determining a target object for performing a target operation on an object (for example, Figure 2 In the embodiment shown, steps 203 and Figure 5 Step 504) in the illustrated embodiment can also be performed in the following manner:

[0143] Firstly, based on the detection results of the human body key points of each image frame to be detected, the movement trajectory of each human body object in the sequence of image frames to be detected is generated.

[0144] Next, a trajectory point in each moving trajectory that is closest to the object is determined as a candidate trajectory point.

[0145] Finally, the target object is determined based on the determined distance between the candidate trajectory point and the object. For example, the human object corresponding to the candidate trajectory point with the smallest distance to the object among the determined candidate trajectory points can be used as the target object. Alternatively, the probability that the human object corresponding to each candidate trajectory point is the target object can be determined based on the distance between each candidate trajectory point and the object, and the human object with the highest probability can be used as the target object.

[0146] It is understandable that in these optional implementations, existing or future-developed face and / or body recognition technologies can be used to distinguish human objects contained in each image frame to be detected, and the movement trajectory of each human object can be obtained in the above manner.

[0147] Furthermore, in some application scenarios of these optional implementation methods, for a human object in an image frame to be detected, if the detection result of the human key point detection of the human object indicates that there are hand key points of the human object in the image frame to be detected, then the hand key points are used as trajectory points on the trajectory of the human object.

[0148] For example, in a certain image frame to be detected in the sequence of image frames to be detected, if a3 is a hand key point representing the hand of the human object A among the key points a1 to a7 of the human object A, then a3 can be used as a trajectory point of the human object A.

[0149] Accordingly, for a human object in an image frame to be detected, if the detection result of the human key point detection of the human object indicates that there is no hand key point of the human object in the image frame to be detected, then according to the key point priority order, the key point with the highest priority among the human key points of the human object existing in the image frame to be detected is selected as the trajectory point on the trajectory of the human object. The specific implementation method of selecting the key point with the highest priority from the human key points of the human object according to the priority order can be performed in the manner described in step 703 above, which will not be repeated here.

[0150] In addition, it is understandable that the probability that the human object determined based on the target image frame is the target object and the probability that the human object determined based on the image frame sequence to be detected is the target object can also be combined to finally determine the target object that has performed the target operation on the object. For example, the probability that the human object determined based on the target image frame is the target object and the probability that the human object determined based on the image frame sequence to be detected is the target object can be weighted in a predetermined weighting manner, and the human object with the maximum probability value in the weighted sum is used as the target object finally determined.

[0151] In the various embodiments, various implementations and various application scenarios described above in the present application, in the human key point detection results, the position of each human key point can be a position in a preset three-dimensional coordinate system. In this way, in the target image frame, the position of each human key point can be represented by a three-dimensional coordinate in the preset three-dimensional coordinate system. Similarly, the trajectory points in the movement trajectory of each human object determined by the sequence of image frames to be detected can also be represented by three-dimensional coordinates in the preset three-dimensional coordinate system.

[0152] For example, the image acquisition device for acquiring image frames may be a depth camera. The depth camera may acquire a planar image (e.g., a color image) and a depth map within its acquisition range, and associate the pixel points in the planar image with the pixel points in the depth map, thereby obtaining not only the color information of any point within its acquisition range but also the depth information of the point.

[0153] In addition, the internal and external parameters of each depth camera used to capture image frames can be calibrated so that the position of each pixel in the depth map captured by each depth camera can correspond to the three-dimensional coordinate position in the same pre-set three-dimensional coordinate system.

[0154] Correspondingly, the position of the object can also be expressed as the position in the same three-dimensional coordinate system.

[0155] For example, a three-dimensional Cartesian coordinate system can be established with a certain point in space as the origin, and the three-dimensional coordinates of each object in the three-dimensional Cartesian coordinate system can be determined. In addition, the internal and external parameters of the depth camera used to collect image frames can be calibrated so that each pixel point in the depth map collected by it has a three-dimensional coordinate in the three-dimensional Cartesian coordinate system.

[0156] In this way, when the sensor detects that an object is moved, the human object that picked up the object can be determined based on the distance between the coordinates (x1, y1, z1) of the object in the three-dimensional coordinate system and the three-dimensional coordinate values ​​of the key points of the human body of each human object in the target image frame. Alternatively, the human object that picked up the object can also be determined based on the distance between the coordinates (x1, y1, z1) of the object in the three-dimensional coordinate system and the coordinates of the trajectory points in the movement trajectory of each human object in the three-dimensional coordinate system.

[0157] The present application also discloses a human-computer interaction system, including: a shelf, an image acquisition device, a sensor and a processor.

[0158] Items are placed on the shelves, and at least one shelf separates preset areas to form a channel.

[0159] An image acquisition device acquires images in at least one channel.

[0160] The sensor detects whether the position of items placed on the shelf changes.

[0161] The processor, in response to detecting a position change of the object, determines a sequence of image frames to be detected based on a detection moment when the position change is detected; performs human key point detection on the image frames to be detected in the sequence of frames to be detected; and determines a target object for performing a target operation on the object based on a detection result of the human key point detection.

[0162] In addition, in the human-computer interaction system of the present application, the processor may also determine the target object by executing the method for determining the target object for performing the target operation behavior as described in any of the above embodiments.

[0163] Optionally, in the above-mentioned human-computer interaction system, the image acquisition device may be a depth camera. The depth camera may acquire a planar image (e.g., a color image) and a depth map within its acquisition range, and associate the pixel points in the planar image with the pixel points in the depth map, thereby obtaining not only the color information of any point within its acquisition range but also the depth information of the point.

[0164] In addition, the internal and external parameters of each depth camera used to capture image frames can be calibrated so that the position of each pixel in the depth map captured by each depth camera can correspond to the three-dimensional coordinate position in the same pre-set three-dimensional coordinate system.

[0165] Further references Figure 8 As an implementation of the methods shown in the above figures, the present application provides an embodiment 800 of a device for determining a target object for performing a target operation behavior. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0166] like Figure 8 As shown, the device for determining a target object for performing a target operation behavior in this embodiment includes a to-be-detected image frame sequence determining unit 801 , a human key point detecting unit 802 , and a target object determining unit 803 .

[0167] The to-be-detected image frame sequence determining unit 801 may be configured to determine the to-be-detected image frame sequence based on a detection moment when the position change is detected in response to detecting the position change of the object.

[0168] The human key point detection unit 802 may be configured to perform human key point detection on the image frames to be detected in the sequence of frames to be detected.

[0169] The target object determination unit 803 may be configured to determine the target object for performing the target operation behavior on the object based on the detection result of the key point detection of the human body.

[0170] In some optional implementations, the image frame sequence determination unit 801 may be further configured to: determine the detection moment when the position change is detected in response to detecting a position change of an object; and arrange image frames whose time difference between the image acquisition moment and the detection moment is less than a preset time length in chronological order of the acquisition moments to form an image frame sequence to be detected.

[0171] In some optional implementations, the human key point detection unit 802 may be further configured to: perform human key point detection on a target image frame whose image acquisition moment is closest to the detection moment in the frame sequence to be detected.

[0172] In some optional implementations, the image frame sequence determination unit 801 may be further configured to: in response to detecting a position change of an object, determine a target image acquisition device for acquiring the image frame sequence to be detected based on the position of the object; and from the image frame sequence acquired by the target image acquisition device, determine that the image frames acquired at the acquisition moment whose time difference with the detection moment is less than a preset time length form the image frame sequence to be detected.

[0173] In some optional implementations, the target object determination unit 803 may be further configured to determine the target object based on the distance between each human object and the object in the detection result of the human key point detection of the target image frame.

[0174] In these optional implementations, the target object determination unit 803 determines the distance between the human object and the object in the target image frame through the following distance determination steps: for the human object in the target image frame, detect whether there is a hand key point indicating the hand of the human object in the target image frame; if so, use the distance between the hand key point of the human object and the object as the distance between the human object and the object.

[0175] In addition, in these optional implementations, the distance determination step also includes: for a human object in the target image frame, if there is no key point indicating the hand of the human object in the target image frame, selecting the key point with the highest priority among the human key points of the human object existing in the target image frame according to the key point priority order; and using the distance between the selected human key point and the object as the distance between the human object and the object.

[0176] In some optional implementations, the target object determination unit 803 may be further configured to: generate a movement trajectory of each human object in the image frame sequence to be detected based on the detection results of the human key point detection of each image frame to be detected; determine the trajectory point in each movement trajectory that is closest to the object as a candidate trajectory point; and determine the target object based on the distance between the determined candidate trajectory point and the object.

[0177] In these optional implementations, for a human object in an image frame to be detected, if the detection result of the human key point detection of the human object indicates that there are hand key points of the human object in the image frame to be detected, then the hand key points are used as trajectory points on the trajectory of the human object.

[0178] In these optional implementations, for a human object in an image frame to be detected, if the detection result of the human key point detection of the human object indicates that there are no hand key points of the human object in the image frame to be detected, then according to the key point priority order, the key point with the highest priority among the human key points of the human object existing in the image frame to be detected is selected as the trajectory point on the trajectory of the human object.

[0179] In some optional implementations, in the detection results of the human body key point detection, the position of each human body key point is the position in a preset three-dimensional coordinate system.

[0180] In some optional implementations, the human key point detection unit 802 can be further configured to: for an image frame to be detected in a sequence of frames to be detected, determine the object area where each human object is located in the image frame to be detected; perform human key point detection on each object area to obtain a detection result of the image frame to be detected.

[0181] Reference below Fig. 9 , which shows a structural diagram of a computer system 900 of an electronic device suitable for implementing a method for determining a target object for performing a target operation behavior according to an embodiment of the present application. Fig. 9 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0182] like Fig. 9 As shown, the computer system 900 includes one or more processors 901 (e.g., CPU), which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 902 or the program loaded from the storage part 906 into the random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the system 900 are also stored. The processor 901, the ROM 902, and the RAM 903 are connected to each other through a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0183] The following components are connected to the I / O interface 905: a storage section 906 including a hard disk and the like; and a communication section 907 including a network interface card such as a LAN card, a modem, and the like. The communication section 907 performs communication processing via a network such as the Internet. A drive 908 is also connected to the I / O interface 905 as needed. A removable medium 909, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, and the like, is installed on the drive 908 as needed so that a computer program read therefrom is installed into the storage section 906 as needed.

[0184] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 907, and / or installed from the removable medium 909. When the computer program is executed by the processing unit 901, the above functions defined in the method of the present application are executed. It should be noted that the computer-readable medium described in the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, - but not limited to - an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, an apparatus or a device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in combination with an instruction execution system, an apparatus or a device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0185] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0186] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0187] The units involved in the embodiments described in the present application may be implemented by software or by hardware. The units described may also be arranged in a processor, for example, may be described as: a processor includes an image frame sequence determination unit to be detected, a human key point detection unit, and a target object determination unit. Among them, the names of these units do not constitute a limitation on the units themselves under certain circumstances. For example, the image frame sequence determination unit to be detected may also be described as "a unit that determines the image frame sequence to be detected in response to detecting a position change of an object and based on the detection moment of detecting the position change."

[0188] As another aspect, the present application also provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by the device, the device: obtains the user image at the current sampling moment; performs face recognition on the obtained user image to determine whether the current user is the same as the user at the previous sampling moment; in response to the current user being the same as the user at the previous sampling moment, the time difference between the current sampling moment and the previous sampling moment is used as the usage time of the current user.

[0189] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above invention concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

Claims

1. A method for determining a target object for performing a target operation behavior, comprising: In response to detecting a change in position of the object, determining a detection time at which the change in position is detected; Arrange the image frames whose time difference between the image acquisition time and the detection time is less than a preset time length in the order of the acquisition time to form a sequence of image frames to be detected; Performing human body key point detection on the image frames to be detected in the sequence of frames to be detected; Based on the detection result of the human body key point detection, a target object for performing a target operation behavior on the object is determined.

2. The method according to claim 1, wherein: The step of performing human body key point detection on the image frames to be detected in the sequence of frames to be detected comprises: Human body key point detection is performed on the target image frame whose image acquisition moment is closest to the detection moment in the frame sequence to be detected.

3. The method according to claim 1, wherein: Also includes: In response to detecting a position change of the object, determining a target image acquisition device for acquiring the sequence of image frames to be detected based on the position of the object; Among them, the image frames whose time difference between the image acquisition moment and the detection moment is less than a preset time length are arranged in the order of acquisition time to form the image frame sequence to be detected, including: from the image frame sequence acquired by the target image acquisition device, the image frames acquired at the acquisition moment whose time difference with the detection moment is less than the preset time length are determined and arranged in the order of acquisition time to form the image frame sequence to be detected.

4. The method according to claim 2, wherein: The step of determining the target object for performing the target operation on the object based on the detection result of the key point detection of the human body includes: The target object is determined based on the distance between each human object and the object in the detection result of the human key point detection of the target image frame.

5. The method according to claim 4, wherein: The distance between the human object in the target image frame and the object is determined by the following distance determination steps: For a human object in the target image frame, detecting whether there is a hand key point indicating a hand of the human object in the target image frame; If so, the distance between the key point of the hand of the human object and the object is used as the distance between the human object and the object.

6. The method according to claim 5, wherein: The distance determination step further comprises: For a human object in the target image frame, if there is no key point indicating a hand of the human object in the target image frame, selecting a key point with the highest priority among the human key points of the human object in the target image frame according to the key point priority order; The distance between the selected human body key point and the object is used as the distance between the human body object and the object.

7. The method according to claim 1, wherein: The step of determining the target object for performing the target operation on the object based on the detection result of the key point detection of the human body includes: Based on the detection results of the human body key points of each of the image frames to be detected, generating the movement trajectory of each human body object in the sequence of image frames to be detected; Determine, in each moving trajectory, a trajectory point that is closest to the object as a candidate trajectory point; The target object is determined according to the distance between the determined candidate trajectory point and the object.

8. The method according to claim 7, wherein: For a human object in an image frame to be detected, if the detection result of the human key point detection of the human object indicates that there are hand key points of the human object in the image frame to be detected, the hand key points are used as trajectory points on the trajectory of the human object.

9. The method according to claim 8, wherein: For a human object in an image frame to be detected, if the detection result of the human key point detection of the human object indicates that the hand key point of the human object does not exist in the image frame to be detected, then according to the key point priority order, the key point with the highest priority among the human key points of the human object existing in the image frame to be detected is selected as the trajectory point on the trajectory of the human object.

10. According to the method of claim 1, in the detection result of the human body key point detection, the position of each human body key point is the position in a preset three-dimensional coordinate system.

11. The method according to claim 1, wherein: The step of performing human body key point detection on the image frames to be detected in the sequence of frames to be detected comprises: For the image frames to be detected in the sequence of frames to be detected, determining the object area where each human body object is located in the image frames to be detected; Perform human key point detection on each object area to obtain the detection result of the image frame to be detected.

12. A human-computer interaction system, comprising: Shelves, on which items are placed, at least one shelf separating preset areas to form aisles; An image acquisition device, which acquires images in at least one channel; A sensor for detecting whether a position of an item placed on the shelf changes; The processor, in response to detecting a position change of an object, determines a detection time when the position change is detected, arranges image frames whose time difference between an image acquisition time and the detection time is less than a preset time length in chronological order of acquisition time to form a sequence of image frames to be detected; performs human key point detection on the image frames to be detected in the sequence of frames to be detected; and determines a target object for performing a target operation on the object based on a detection result of the human key point detection.

13. The human-computer interaction system according to claim 12, wherein: The image acquisition device is a depth camera.

14. A device for determining a target object for performing a target operation behavior, comprising: a unit for determining a sequence of image frames to be detected, configured to, in response to detecting a position change of an object, determine a detection time when the position change is detected, and arrange image frames whose time difference between an image acquisition time and the detection time is less than a preset time length in a chronological order of acquisition times to form a sequence of image frames to be detected; A human key point detection unit is configured to perform human key point detection on the image frames to be detected in the sequence of frames to be detected; The target object determination unit is configured to determine the target object for performing the target operation behavior on the object based on the detection result of the key point detection of the human body.

15. An electronic device, comprising: interface; one or more processors; a memory having one or more programs stored therein, And one or more processors, operatively connected to the above-mentioned interface and the above-mentioned memory, for executing the method according to any one of claims 1-11.

16. A computer-readable storage medium having a computer program stored thereon, wherein: When the above computer program is executed by one or more processors, the one or more processors are enabled to execute the method according to any one of claims 1 to 11.

Images