Mechanical arm interaction method and device based on visual identification and electronic equipment

Abstract

The invention provides a mechanical arm interaction method and device based on visual recognition and electronic equipment, and the method comprises the steps that an image collection data set is obtained, the image collection data set comprises m pieces of image collection data, one piece of image collection data comprises images collected by all cameras in n cameras at one image collection moment, and m and n are positive integers; based on the image acquisition data set, target mechanical arm action mapping indication data corresponding to the target object is determined; wherein the target object is the object with the maximum object area indication value in the at least one object, and the object area indication value of one object is used for indicating the size of the area occupied by the corresponding object; and based on the target mechanical arm action mapping indication data, target mechanical arm actions are determined, and the mechanical arm is controlled to execute the target mechanical arm actions. According to the embodiment of the invention, the interactivity of the mechanical arm can be improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and robot control technology, and in particular to a robot arm interaction method, device and electronic equipment based on visual recognition. Background Art

[0002] With the continuous development of robot technology, robots are gradually moving from traditional industrial applications to areas closer to people's lives, such as services and entertainment. At present, robot dance, as a novel form of performance, has attracted more and more attention; among them, robotic arms, as a common basic form of robots, can realize robot dancing. However, in related technologies, the implementation of robot dance mainly relies on pre-programmed control of robots to complete a series of fixed movements. It is impossible to adjust the movements in real time according to changes in the external environment, lacks interactivity, and it is difficult to bring a better viewing experience to the audience. Based on this, there is currently no good solution to how to improve the interactivity of robotic arms. Summary of the invention

[0003] In view of this, the embodiments of the present invention provide a robotic arm interaction method, device and electronic device based on visual recognition to solve the problems such as the lack of interactivity of the robotic arm involved in the related technology; that is, the embodiments of the present invention can determine the target robotic arm movement in real time through the image acquisition data set, and control the robotic arm to perform the target robotic arm movement, so as to adjust the movement of the robotic arm according to changes in the external environment, thereby effectively improving the interactivity of the robotic arm and bringing a better viewing experience to the audience.

[0004] According to one aspect of an embodiment of the present invention, a robot arm interaction method based on visual recognition is provided, the method comprising:

[0005] Acquire an image acquisition data set, wherein the image acquisition data set includes m image acquisition data, one image acquisition data includes an image acquired by each camera of the n cameras at an image acquisition time, and m and n are both positive integers;

[0006] Based on the image acquisition data set, determining the target manipulator motion mapping indication data corresponding to the target object; wherein the target object is an object with the largest object area indication value among at least one object, and the object area indication value of an object is used to indicate the size of the area occupied by the corresponding object;

[0007] Based on the target robot arm motion mapping indication data, a target robot arm motion is determined, and the robot arm is controlled to perform the target robot arm motion.

[0008] According to another aspect of an embodiment of the present invention, a robotic arm interaction device based on visual recognition is provided, the device comprising:

[0009] An image acquisition module is used to acquire an image acquisition data set, wherein the image acquisition data set includes m image acquisition data, one image acquisition data includes an image acquired by each camera of n cameras at an image acquisition time, and m and n are both positive integers;

[0010] An image analysis and processing module, used to determine the target manipulator motion mapping indication data corresponding to the target object based on the image acquisition data set; wherein the target object is an object with the largest object area indication value among at least one object, and the object area indication value of an object is used to indicate the size of the area occupied by the corresponding object;

[0011] An action generation module, used for determining a target manipulator action based on the target manipulator action mapping indication data;

[0012] The robot arm control module is used to control the robot arm to perform the target robot arm action.

[0013] According to another aspect of an embodiment of the present invention, an electronic device is provided, comprising a processor and a memory storing a program, wherein the program comprises instructions, which, when executed by the processor, enable the processor to perform the above-mentioned method.

[0014] According to another aspect of an embodiment of the present invention, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the above-mentioned method.

[0015] After acquiring the image acquisition data set, the embodiment of the present invention can determine the target mechanical arm motion mapping indication data corresponding to the target object based on the image acquisition data set, the image acquisition data set includes m image acquisition data, one image acquisition data includes the image acquired by each camera of n cameras at one image acquisition time, and m and n are both positive integers; wherein the target object is the object with the largest object area indication value among at least one object, and the object area indication value of an object is used to indicate the size of the area occupied by the corresponding object. Based on this, the target mechanical arm motion can be determined based on the target mechanical arm motion mapping indication data, and the mechanical arm can be controlled to perform the target mechanical arm motion. It can be seen that the embodiment of the present invention can determine the target mechanical arm motion in real time through the image acquisition data set, and control the mechanical arm to perform the target mechanical arm motion, so as to adjust the motion of the mechanical arm according to the changes in the external environment, thereby realizing a mechanical arm interaction method based on visual recognition, thereby effectively improving the interactivity of the mechanical arm, and providing a better viewing experience for the audience. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Further details, features and advantages of the invention are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0017] Figure 1 A schematic flow chart of a robot arm interaction method based on visual recognition according to an exemplary embodiment of the present invention is shown;

[0018] Figure 2 A schematic flow chart of another robotic arm interaction method based on visual recognition according to an exemplary embodiment of the present invention is shown;

[0019] Figure 3 A schematic block diagram of a robotic arm interaction device based on visual recognition according to an exemplary embodiment of the present invention is shown;

[0020] Figure 4 A block diagram of an exemplary electronic device that can be used to implement an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0021] Embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments of the present invention are only for exemplary purposes and are not intended to limit the scope of protection of the present invention.

[0022] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0023] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the following description. It should be noted that the concepts of "first", "second", etc. mentioned in the present invention are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0024] It should be noted that the modifications of "one" and "plurality" mentioned in the present invention are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0025] The names of the messages or information exchanged between multiple devices in the embodiments of the present invention are only used for illustrative purposes, and are not used to limit the scope of these messages or information.

[0026] It should be noted that the execution subject of the visual recognition-based robot arm interaction method provided in the embodiment of the present invention may be one or more electronic devices, which is not limited in the embodiment of the present invention; wherein, the electronic device may be a terminal (i.e., a client) or a server, then when the execution subject includes multiple electronic devices, and the multiple electronic devices include at least one terminal and at least one server, the visual recognition-based robot arm interaction method provided in the embodiment of the present invention may be jointly executed by the terminal and the server. Accordingly, the terminal mentioned here may include but is not limited to: a robot, a tablet computer, a laptop computer, a desktop computer, and the like. The server mentioned here may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing (cloud computing), cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and basic cloud computing services such as big data and artificial intelligence platforms, and the like. Exemplarily, when the executor of the robotic arm interaction method based on visual recognition is an electronic device (such as a robot), all steps of the robotic arm interaction method based on visual recognition can be completed by the electronic device; when the executor of the robotic arm interaction method based on visual recognition is multiple electronic devices (such as a robot and electronic devices other than the robot), the electronic devices can interact with each other, such as electronic devices other than the electronic device used to control the robotic arm to perform actions (such as a robot) can determine certain data to transmit the determined corresponding data to the electronic device used to control the robotic arm to perform actions, thereby enabling the electronic device used to control the robotic arm to perform actions to perform subsequent operation steps until the robotic arm is controlled to perform corresponding actions, and so on; the embodiments of the present invention are not limited to this.

[0027] Based on the above description, an embodiment of the present invention proposes a robot arm interaction method based on visual recognition, which can be executed by one or more electronic devices mentioned above. For the sake of convenience, the following description will be given by taking an electronic device (such as a robot) executing the robot arm interaction method based on visual recognition as an example; Figure 1 As shown, the robot arm interaction method based on visual recognition may include the following steps S101-S103:

[0028] S101, obtaining an image acquisition data set, where the image acquisition data set includes m image acquisition data, where one image acquisition data set includes an image acquired by each camera among n cameras at one image acquisition moment, and both m and n are positive integers.

[0029] Among them, the above-mentioned n cameras can be used for image acquisition; optionally, the electronic device may include n cameras, that is, the electronic device may be installed with n cameras, and the electronic device may include an image acquisition module to acquire images through the image acquisition module; or, the electronic device may establish a connection with the n cameras (such as a wired or wireless connection, etc.), then the n cameras can transmit the acquired images to the electronic device, so that the electronic device can obtain the images acquired by the n cameras, etc.; the embodiment of the present invention is not limited to this. It should be noted that the time between the n cameras is synchronized, so that in the subsequent image processing process, the images acquired by each camera at the same acquisition time can be determined; optionally, the embodiment of the present invention can be synchronized by hardware, in which case, it is preferred to use a hardware synchronization device to ensure that multiple cameras acquire images at the same time, such as synchronizing the acquisition clocks of all cameras through an external trigger signal, so that they can start and end image acquisition at the same time; or, it can be synchronized by software, in which case, a timestamp can be added to the image data acquired by each camera to record the time of image acquisition, and then in the subsequent processing process, the images are aligned according to the timestamp, etc.; the embodiment of the present invention is not limited to this.

[0030] Optionally, the image acquisition data set may be obtained in a manner including but not limited to at least one of the following:

[0031] The first acquisition method: each image acquisition data in the m image acquisition data (i.e., image acquisition data at m image acquisition moments) is image acquisition data acquired by the n cameras in real time, i.e., newly acquired image acquisition data; in this case, the electronic device can add the image acquired by the n cameras at each image acquisition moment in the m image acquisition moments to the image acquisition data set to acquire the image acquisition data set, and at this time, the image acquisition data in the image acquisition data set are all newly acquired image acquisition data.

[0032] The second acquisition method: the m image acquisition data include at least one real-time acquired image acquisition data and at least one historical image acquisition data; in this case, the electronic device can obtain the image (i.e., the newly acquired image) acquired by the n cameras at each real-time image acquisition moment in at least one real-time image acquisition moment, and determine at least one historical image acquisition data closest to the current system time from the historical data (which may include the image acquisition data previously acquired by the n cameras), so as to add the at least one historical image acquisition data and the at least one real-time acquired image data to the image acquisition data set respectively, so as to realize the acquisition of the image acquisition data set, and so on.

[0033] When the value of n is greater than 1, the electronic device needs to perform spatial calibration and data fusion to convert the coordinates of each camera coordinate system into integrated world coordinates and image coordinates. Optionally, each camera can be calibrated with its own internal parameters to obtain its internal parameters. For example, taking the i-th camera among n cameras as an example, the internal parameters may include focal length (f i ), optical center coordinates and pixel size Etc., which can be done by traditional checkerboard calibration methods; using these internal parameters, the image coordinates can be converted to coordinates in the camera coordinate system to prepare for subsequent triangulation. Optionally, the electronic device can also determine the relative position relationship between the n cameras, that is, the external parameters; the rotation matrix (R) of each camera relative to a common reference coordinate system (such as the world coordinate system) can be obtained through precise measurement tools and specific calibration processes. i ) and the translation vector (T i ), this step can be implemented using an extended form of Zhang Zhengyou's calibration method or a multi-camera calibration method based on special calibration objects to ensure accurate relative position information, so that the position of the face in three-dimensional space can be accurately estimated through triangulation.

[0034] Optional, for camera C i , the internal parameter matrix K i It can be defined as shown in formula 1.1:

[0035]

[0036] Based on this, suppose a point D(X w ,Y w ,Z w ) In the world coordinate system, it is at camera C i Coordinate D in the coordinate system i (X i ,Y i ,Z i ) can be calculated by formula 1.2:

[0037]

[0038] Then, we can use formula 1.3 to get the internal parameter matrix K i Convert the coordinates in the camera coordinate system to image coordinates (u i ,v i ),:

[0039]

[0040] S102, based on the image acquisition data set, determine the target robot arm motion mapping indication data corresponding to the target object; wherein the target object is the object with the largest object area indication value among at least one object, and the object area indication value of an object is used to indicate the size of the area occupied by the corresponding object.

[0041] Optionally, an object can be a person or an object, which is not limited in the embodiment of the present invention. Optionally, when an object is a person, the object area indication value of the object can be the percentage of the object body area in the entire image (such as the total number of pixels in the object body area divided by the total number of pixels in the image), or the percentage of all areas of the object in the entire image. When the size of the image captured by each camera is the same (that is, the total number of pixels in the image is the same), it can also be the total number of pixels in the body area of ​​the object, and so on; the embodiment of the present invention does not limit this. Optionally, when an object is an object, the object area indication value of the object can be the percentage of the object area (that is, the area where the object is located) in the entire image (that is, the percentage of the object area in the corresponding image, such as the total number of pixels in the object area divided by the total number of pixels in the image where the object is located), and when the size of the image captured by each camera is the same, it can also be the total number of pixels in the object area, and so on; the embodiment of the present invention does not limit this. Optionally, when the object area indication value of an object is the percentage of the corresponding area of ​​the corresponding object in the image, the object area indication value of an object can also be called the visual area percentage of the corresponding object.

[0042] Based on this, the object area indication value of an object is positively correlated with the size of the area occupied by the corresponding object (i.e., the area occupied by the corresponding object in the corresponding image); that is, the larger the object area indication value of an object is, the larger the area occupied by the object is, and the smaller the object area indication value of an object is, the smaller the area occupied by the object is.

[0043] Then, accordingly, the electronic device can determine the object with the largest area from at least one object, so as to use the object with the largest area as the target object; that is, the electronic device can select the object with the largest object area indication value from at least one object as the target object. Optionally, at least one object may include all objects in each image in the image acquisition data set; or, at least one object may include objects that exist in the image acquisition data at multiple consecutive image acquisition moments in the image acquisition data set; or, at least one object may include all objects in the image acquisition data at any image acquisition moment in m image acquisition moments, and so on; the embodiment of the present invention is not limited to this. Optionally, the number of target objects can be one or more, and the embodiment of the present invention is not limited to this; illustratively, the electronic device can also use all objects in the image acquisition data at each image acquisition moment as at least one object to determine the target object respectively; in this case, the target object at each image acquisition moment can be determined respectively, and so on. Among them, the specific implementation method of determining the target object can be shown as follows, and the embodiment of the present invention will not be repeated here.

[0044] S103, determining a target robotic arm motion based on the target robotic arm motion mapping indication data, and controlling the robotic arm to execute the target robotic arm motion.

[0045] Optionally, the number of target robotic arm actions may be one or more, which is not limited in the embodiments of the present invention; wherein, a target robotic arm action may be determined based on a target robotic arm action mapping indication data, then when the target robotic arm action mapping indication data changes, that is, when the number of target robotic arm action mapping indication data is multiple, multiple target robotic arm actions may be determined, that is, the target robotic arm actions corresponding to each target robotic arm action mapping indication data may be determined separately, and so on.

[0046] After acquiring the image acquisition data set, the embodiment of the present invention can determine the target mechanical arm motion mapping indication data corresponding to the target object based on the image acquisition data set, the image acquisition data set includes m image acquisition data, one image acquisition data includes the image acquired by each camera of n cameras at one image acquisition time, and m and n are both positive integers; wherein the target object is the object with the largest object area indication value among at least one object, and the object area indication value of an object is used to indicate the size of the area occupied by the corresponding object. Based on this, the target mechanical arm motion can be determined based on the target mechanical arm motion mapping indication data, and the mechanical arm can be controlled to perform the target mechanical arm motion. It can be seen that the embodiment of the present invention can determine the target mechanical arm motion in real time through the image acquisition data set, and control the mechanical arm to perform the target mechanical arm motion, so as to adjust the motion of the mechanical arm according to the changes in the external environment, thereby realizing a mechanical arm interaction method based on visual recognition, thereby effectively improving the interactivity of the mechanical arm, and providing a better viewing experience for the audience.

[0047] Based on the above description, the embodiment of the present invention further proposes another robot arm interaction method based on visual recognition. Accordingly, the robot arm interaction method based on visual recognition can be executed by one or more electronic devices mentioned above. For the sake of convenience, the following description will be given by taking an electronic device (such as a robot arm) executing the robot arm interaction method based on visual recognition as an example; please refer to Figure 2 The robot arm interaction method based on visual recognition may include the following steps S201-S205:

[0048] S201, obtaining an image acquisition data set, where the image acquisition data set includes m image acquisition data, and one image acquisition data set includes an image acquired by each camera among n cameras at an image acquisition moment.

[0049] S202, perform person detection on each image in the image acquisition data set to obtain person detection data for each image. The person detection data of an image includes a person segmentation mask image of the corresponding image and person key point indication data of each person in the corresponding image. The person segmentation mask image of an image includes a segmentation mask of each person in the corresponding image.

[0050] Optionally, the electronic device may use algorithms such as PoseNet or MoveNet (a technical solution for posture detection) to perform person detection on each image in the image acquisition data set, which is not limited in the embodiments of the present invention. Among them, the PoseNet algorithm can generate a specific segmentation mask of the person (distinguishing different parts of the human body, that is, different masks for different parts of the human body) for calculating the object area indication value, such as the proportion of the number of pixels in the body area of ​​the person in the image; and can also identify key points of the human body for posture recognition of the person.

[0051] Optionally, the key point indication data of a character may include the position information of the key points of the corresponding character's body, so that the changes in the coordinates of these key points can be analyzed to obtain the character's motion information, such as the body's inclination angle, the direction and amplitude of limb movement (i.e., the angle between the arm and the torso), etc.

[0052] S203, determining a target object based on the person segmentation mask images of each image; wherein the target object is an object having a maximum object area indication value among at least one object, and the object area indication value of an object is used to indicate the size of an area occupied by the corresponding object.

[0053] Among them, the person segmentation mask image of an image may include the segmentation mask of each person in the corresponding image; based on this, for any person in the image acquisition data set (i.e., any person among all the people indicated by the image acquisition data set), the object area indication value of any person can be determined based on the segmentation mask of any person, and a person can be an object. Exemplarily, when the object area indication value of a person is the percentage of the body area of ​​the corresponding person in the image, the number of pixels in the body area of ​​any person can be determined based on the segmentation mask of any person, so as to use the ratio between the number of pixels in the body area of ​​any person and the number of pixels in the image where any person is located as the object area indication value of any person, and so on.

[0054] Optionally, the electronic device may also perform object detection on each image in the image acquisition data set to obtain object detection data of each image, and the object detection data of one image includes the object segmentation mask image of the corresponding image. The object segmentation mask image of one image may include the segmentation mask of each object in the corresponding image; accordingly, for any object in the image acquisition data set (i.e., any object among all objects indicated by the image acquisition data set), the object area indication value of any object may be determined based on the segmentation mask of any object, and an object (also referred to as an object) may be an object; exemplarily, when the object area indication value of an object is the percentage of the corresponding object area occupied by the image, the number of pixels in the area of ​​any object may be determined based on the segmentation mask of any object, so as to use the ratio between the number of pixels in the area of ​​any object and the number of pixels in the image where any object is located as the object area indication value of any object, and so on. Optionally, when performing object detection on any image, the object semantic information of each object in any image may also be obtained (which can be used to indicate the category of the object, such as cat, dog, etc.).

[0055] Optionally, the electronic device may use an object recognition deep learning model to perform object detection on each image separately to identify the object in the image and generate a segmentation mask of the corresponding object. Optionally, the object recognition deep learning model may be any pre-trained object recognition deep learning model, which is not limited in the embodiment of the present invention; illustratively, the object recognition deep learning model may be a SAM (Segment Anything Model) model (an image segmentation task model), and so on.

[0056] Based on this, when determining the target object based on the person segmentation mask image of each image, the target object can be determined based on the person segmentation mask image and the object segmentation mask image of each image, and the target object is a person or an object. Optionally, for any object in any image in the image acquisition data set (i.e., any object indicated by any image), when any object is a person, the segmentation mask of any object can be determined from the person segmentation mask image of the image where any object is located, and when any object is an object, the segmentation mask of any object can be determined from the object segmentation mask image of the image where any object is located, and then the object area indication value of any object can be determined based on the segmentation mask of any object.

[0057] In one embodiment, when determining the target object based on the person segmentation mask image and the object segmentation mask image of each image, the electronic device can determine the object area indication value of each object in each image based on the person segmentation mask image and the object segmentation mask image of each image; then, based on the object area indication value of each object in each image, the pending object with the largest object area indication value at each image acquisition moment in the m image acquisition moments can be determined respectively (the pending object with the largest object area indication value at one image acquisition moment can be the object with the largest object area indication value among all objects indicated by the images acquired by each camera at the corresponding image acquisition moment), so as to determine the target object from the m pending objects by using the voting method, and the target object can be the pending object with the largest number of identical pending objects among the m pending objects. Optionally, when both objects are objects, it can be determined whether they are the same object based on the object semantic information of the two objects, and when both objects are persons, it can be determined whether they are the same person based on the similarity of the face regions of the two objects, and so on; the embodiment of the present invention is not limited to this.

[0058] In another embodiment, for any image acquisition moment among the m image acquisition moments, the electronic device may first determine a fusion object set at any image acquisition moment from all objects in the images acquired by each camera at any image acquisition moment. A fusion object in the fusion object set may correspond to one or more objects in all objects in the images acquired by each camera at any image acquisition moment. That is to say, the same object (i.e., the same object) in all objects in the images acquired by each camera at any image acquisition moment may be taken as a fusion object. For example, it may be determined whether they are the same object by object semantic information or face area. Then, for any fusion object in the fusion object set, the object area indication value of any fusion object may be determined according to the weight of the camera that acquires the image of each object corresponding to any fusion object and the object area indication value of each object corresponding to any fusion object. In this case, the object area indication value of any fusion object may be determined by a weighted average method or a Bayesian reasoning method. Furthermore, after obtaining the fusion object set at each image acquisition moment, the fusion object with the largest object area indication value at each image acquisition moment can be determined from the fusion object set at each image acquisition moment as the pending object at the corresponding image acquisition moment, so as to determine m pending objects, and then determine the target object from the m pending objects by voting.

[0059] In another embodiment, the electronic device may also respectively regard the pending object at each image acquisition moment as the target object at the corresponding image acquisition moment, that is, respectively regard them as a target object, and the number of target objects may be m at this time; or, the pending object at the mth image acquisition moment may be regarded as the target object, and the target object may be determined in the environment at the most recent moment, and so on; the embodiment of the present invention is not limited to this.

[0060] Accordingly, when the target object is a person, the electronic device can trigger the execution of the following person key point indication data based on the target object to determine the target robotic arm action mapping indication data corresponding to the target object; when the target object is an object, the object semantic information of the target object can be determined, and the object semantic information of the target object can be added to the target robotic arm action mapping indication data corresponding to the target object, one object semantic information corresponds to one robotic arm action.

[0061] Optionally, when there are multiple target objects, the target manipulator action mapping indication data corresponding to each of the multiple target objects may be determined respectively; optionally, the target manipulator action mapping indication data corresponding to different target objects may be the same or different, which is not limited in the embodiment of the present invention. For ease of explanation, the following description will be made by taking one target object as an example.

[0062] S204: Determine target mechanical arm motion mapping indication data corresponding to the target object based on the character key point indication data of the target object.

[0063] In an embodiment of the present invention, the electronic device can determine the human body posture action of the target object based on the human body key point indication data of the target object, and add the human body posture action identifier of the human body posture action of the target object to the target mechanical arm action mapping indication data corresponding to the target object, so as to determine the target mechanical arm action. Optionally, the number of human body key point indication data of the target object can be one or more; optionally, the electronic device can determine the human body key point indication data of all objects that are the same object as the target object from the human body key point indication data of each character in each image (the characters whose similarity of face areas in different images is greater than a preset similarity threshold can be the same character) to obtain the human body key point indication data of the target object at each image acquisition moment in one or more image acquisition moments (i.e., multiple human body key point indication data of the target object); wherein, one human body key point indication data can be used to indicate one human body posture. Optionally, one or more human body key point indication data (i.e., human body key point coordinates) of the target object can be expressed as a human body key point indication data sequence {P j}, where j represents the sequence number of the jth human posture. Optionally, the electronic device may determine the human posture action of the target object, such as horizontal waving or vertical waving, etc., based on the changes in the multiple human key point indication data of the target object; or, a specific human posture in the human posture indicated by one or more human key point indication data of the target object may be used as the human posture action of the target object, and so on; the embodiment of the present invention is not limited to this. Optionally, the specific human posture may be any human posture in the human posture set; optionally, the human posture set may be set according to experience or actual needs, which is not limited to this in the embodiment of the present invention. Based on this, the human posture action of the target object may be static or dynamic, which is not limited to this in the embodiment of the present invention.

[0064] Optionally, the electronic device may also determine the action guidance data of the target object based on the character key point indication data of the target object, and add the action guidance data to the target robotic arm action mapping indication data; wherein the action guidance data includes at least one of the following: the character's arm amplitude (i.e., the angle between the arm and the body trunk), the character's action speed, and the character's action frequency, and the action guidance data supports the determination of robotic arm action parameter data of the robotic arm, so that the robotic arm performs the target robotic arm action according to the robotic arm action parameter data. It should be noted that the embodiments of the present invention do not limit the specific method for determining the action guidance data; for example, based on the character key point indication data of the target object, the angle between the upper arm and the torso of the target object can be determined as the character arm amplitude of the target object, or the angle between the lower arm and the torso of the target object can be determined, so as to perform weighted summation on the angle between the upper arm and the torso and the angle between the lower arm and the torso to obtain the character arm amplitude of the target object, or when the character arm amplitudes indicated by each character key point indication data of the target object are different, the maximum value of the character arm amplitudes can be used as the character arm amplitude of the target object, or the character arm amplitudes indicated by each character key point indication data of the target object can be respectively used as the character arm amplitude of the target object at the corresponding image acquisition time. For example, based on the target object's key point indication data, the maximum position change distance and position change time (i.e., the time it takes to generate the maximum position change distance) of the target key point with the largest position information change can be determined, and the ratio between the maximum position change distance and the position change time of the target key point can be used as the character's action speed, or the ratio between the maximum position change distance and the corresponding position change time of the designated key point can be used as the character's action speed, etc.; for example, the maximum value of the number of action changes can be used as the target object's character action frequency (i.e., action change frequency), such as when swinging left and swinging right alternately, the change from swinging left to swinging right or from swinging right to swinging left can be one action change, etc. Optionally, the designated key point can be set according to experience or according to actual needs, and the embodiment of the present invention is not limited to this.

[0065] S205 , determining a target robotic arm motion based on the target robotic arm motion mapping indication data, and controlling the robotic arm to execute the target robotic arm motion.

[0066] In an embodiment of the present invention, when the target object is a person, the target mechanical arm motion mapping indication data includes a human posture motion identifier; when the target object is an object, the target mechanical arm motion mapping indication data includes object semantic information. Based on this, when determining the target mechanical arm motion based on the target mechanical arm motion mapping indication data, when the target mechanical arm motion mapping indication data includes a human posture motion identifier, based on the human posture motion identifier in the target mechanical arm motion mapping indication data, a target mechanical arm motion identifier that matches the human posture motion identifier in the target mechanical arm motion mapping indication data is determined from the first mechanical arm motion set, and the target mechanical arm motion identifier is used to indicate the target mechanical motion, so as to determine the target mechanical arm motion. Among them, a human posture motion identifier can be used to indicate a human posture motion; optionally, a human posture motion identifier can be any human posture motion identifier in the human posture motion identifier set; optionally, a human posture motion identifier can be the name of a human posture motion, or can be the number of a human posture motion, etc., and the embodiment of the present invention does not limit this. Accordingly, a robot action identifier can be used to indicate a robot action; optionally, a robot action identifier can be a name of a robot action, or a number of a robot action, etc., which is not limited in the embodiment of the present invention. Optionally, the human body posture action identifier set can be set according to experience or actual needs, which is not limited in the embodiment of the present invention.

[0067] Optionally, the first robotic arm action set may include the robotic arm action identifiers (i.e., first robotic arm action identifiers) of each first robotic arm action in at least one first robotic arm action; optionally, the first robotic arm action set may be pre-compiled according to experience or actual needs, which is not limited in the embodiments of the present invention. Optionally, the human body posture action identifier set may be mapped to the first robotic arm action set, that is, any human body posture action identifier in the human body posture action identifier set may be matched with a first robotic arm action identifier in the first robotic arm action set, so that a target robotic arm action identifier that matches the human body posture action identifier in the target robotic arm action mapping indication data may be determined from the first robotic arm action set.

[0068] Correspondingly, when the target manipulator action mapping indication data includes an item semantic information, based on the item semantic information in the target manipulator action mapping indication data, a target manipulator action identifier that matches the item semantic information in the target manipulator action mapping indication data is determined from the second manipulator action set. Optionally, the second manipulator action set may include the manipulator action identifiers (i.e., second manipulator action identifiers) of each second manipulator action in at least one second manipulator action; optionally, the second manipulator action set may be pre-compiled according to experience or actual needs, which is not limited in the embodiment of the present invention. Optionally, each item semantic information in the item semantic information set may correspond to (i.e., match) a second manipulator action identifier in the second manipulator action set, that is, there is a mapping relationship between the item semantic information set and the second manipulator action set; optionally, the item semantic information set may be set according to experience or actual needs, which is not limited in the embodiment of the present invention; wherein, the item semantic information in the target manipulator action mapping indication data (i.e., the item semantic information of the target object) may be any item semantic information in the item semantic information set. Optionally, the second robot arm action set may be the same as the first robot arm action set (in this case, a robot arm action set may have a mapping relationship with the object semantic information set and the human body posture action identification set respectively), or may be different from the first robot arm action set, and the embodiment of the present invention does not limit this. For example, when the object semantic information is a cat or a toy cat, the robot arm can make gentle movements imitating a cat; when the object semantic information is a robot or a toy robot, the robot arm can make a stuttering movement imitating a robot, and so on.

[0069] Optionally, when the target object is a person, the target robotic arm motion mapping indication data may include motion guidance data of the target object, the motion guidance data including at least one of the following: the arm amplitude of the person, the motion speed of the person and the motion frequency of the person; in this case, the electronic device may also determine the pending robotic arm motion parameter data of the robotic arm based on the motion guidance data, the pending robotic arm motion parameter data including at least one of the following: the pending robotic arm motion amplitude, the pending robotic arm motion frequency and the pending robotic arm motion speed; wherein, the pending robotic arm motion amplitude is determined based on the arm amplitude of the person, the pending robotic arm motion frequency is determined based on the motion frequency of the person, and the pending robotic arm motion speed is determined based on the motion speed of the person; accordingly, the robotic arm motion parameter data of the robotic arm can be determined based on the pending robotic arm motion parameter data. Based on this, when controlling the robot arm to perform the target robot arm action, the robot arm can be controlled to perform the target robot arm action according to the robot arm action parameter data; that is, after determining the robot arm action parameter data, the robot arm can be controlled to perform the target robot arm action according to the robot arm action parameter data, such as making the speed, frequency and amplitude of the target robot arm action the same as the corresponding data indicated by the robot arm action parameter data. The target robot arm action can also be referred to as the robot arm action content.

[0070] Optionally, when determining the pending mechanical arm motion parameter data of the mechanical arm based on the motion guidance data, the electronic device may determine the pending mechanical arm motion amplitude based on the human arm amplitude and the original mechanical arm motion amplitude, such as the pending mechanical arm motion amplitude B θ = B0×(θ / 90), B0 represents the original motion amplitude of the robot arm (i.e., the original motion amplitude of the robot arm), θ represents the amplitude of the character's arm, and the larger the amplitude of the character's arm, the larger the motion amplitude of the pending robot arm; and / or, the motion frequency of the pending robot arm can be determined based on the motion frequency of the character and the original motion frequency of the robot arm, such as the motion frequency of the pending robot arm F k =F0×(F pk / F p0 ), F0 represents the original action frequency of the robot arm, F pk Indicates the frequency of character actions, F p0 represents the standard frequency of the character waving his arms. The higher the character's action frequency, the higher the frequency of the pending mechanical arm's action. And / or, the pending mechanical arm's action speed can be determined based on the character's action speed and the original movement speed of the mechanical arm, such as the pending mechanical arm's action speed V k =V0×(V pk / V p0 ), V0 represents the original moving speed of the robot arm, V pk Indicates the character's movement speed, V p0It indicates the standard moving speed of the character. The faster the character moves, the faster the to-be-determined mechanical arm moves, etc. Optionally, the standard frequency of the character waving his arms and the standard moving speed of the character can be set according to experience or actual needs, which is not limited in the embodiment of the present invention.

[0071] In one embodiment, when determining the robotic arm motion parameter data of the robotic arm based on the pending robotic arm motion parameter data, the electronic device may perform face detection on each image in the image acquisition data set to obtain the face detection results of each image. In a specific implementation, the electronic device may use any pre-trained face detection model (such as a face detection model based on a convolutional neural network, etc., which is trained with a large amount of face image data and can recognize the face area in the image) to perform face detection on each image; optionally, the face detection result may include face position information, such as expressed in rectangular frame coordinates. In another specific implementation, for any image in the image acquisition data set, the electronic device may use a face detection algorithm based on Haar features, slide a window on any image to extract the Haar features of any image, and use the trained AdaBoost classifier to determine whether the window area is a face area to record the position of the face, such as expressed in image coordinates, such as the coordinates of the upper left corner. and the lower right corner coordinates and size information (width w i and height h i ), and extract the Haar feature vector of the face For subsequent feature matching, i represents the i-th camera C i Suppose from camera C i The Haar feature vector of the face area detected in is (d is the feature vector dimension).

[0072] Then, based on the face detection results of each image, a headcount result can be determined, and based on the headcount result, the robot arm motion adjustment data of the robot arm can be determined; the robot arm motion adjustment data includes the adjustment of the robot arm motion amplitude and / or the adjustment of the robot arm motion frequency. Optionally, when determining the headcount result based on the face detection results of each image, the electronic device can determine the number of faces at each image acquisition moment based on the face detection results of each image, and perform mean calculation on the number of faces at each image acquisition moment to obtain the headcount result; or, the number of faces at the mth image acquisition moment can be used as the headcount result; or, the number of faces at each image acquisition moment can be used as a headcount result, and so on; the embodiment of the present invention is not limited to this.

[0073] In a specific implementation, for any image acquisition moment among the m image acquisition moments, the electronic device can determine the position and size information of the face area in the image captured by each camera at any image acquisition moment based on the face detection results of the image captured by each camera at any image acquisition moment, and perform preliminary screening to remove obvious mismatches; for example, the positions are too far apart (that is, the position distance between any two human face areas is greater than a preset distance threshold, and they can be considered not to be the same face), the size ratio is extremely different (such as the size ratio between any two human face areas is greater than a preset size ratio threshold, etc., and they can be considered not to be the same face), etc.; optionally, the preset distance threshold and the preset size ratio threshold can be set according to experience or according to actual needs, and the embodiments of the present invention are not limited to this. Then, for any remaining face region pair (i.e., any face region pair in all face region pairs in the image captured by each camera at any image capture time except for the face region pairs that are obviously mismatched, and the two face regions in a face region pair are face regions in images captured by different cameras), the similarity of any face region pair can be calculated (such as the similarity of Haar feature vectors, measured using the cosine similarity function), so as to determine whether the two faces indicated by any face region pair are the same face based on the similarity. Among them, camera C i and C j The cosine similarity calculation of the feature vectors between the face regions in the collected images is shown in Formula 2.1:

[0074]

[0075] Where S in Formula 2.1 represents cosine similarity. Optionally, when the similarity (such as cosine similarity) is greater than a preset similarity threshold, triangulation can be further used to verify whether it is the same face. Optionally, assume that camera C i and C j The optical centers are O i and O j , the image planes are π i and π j For the possible matching face regions detected in the two camera images (whose image coordinates are known), the three-dimensional coordinates (X and Y) of the face in the world coordinate system are calculated based on the intrinsic and extrinsic parameters of the cameras and the triangle similarity principle. w ,Y w ,Z w ). Optionally, the specific calculation method can be to establish a similar triangle relationship consisting of the optical center, the pixel point on the image plane and the space point, and solve the three-dimensional coordinates using the known internal and external parameters. Suppose the camera C i The coordinates of the optical center in the world coordinate system are Camera Cj The optical center coordinates are For camera C i The pixel coordinates of the face area detected in the collected image (u i ,v i ) (such as including the coordinates of all pixels in the corresponding face area) and at camera C j The pixel coordinates of the face area that may be matched in the collected image (u j ,v j ), according to the internal parameter matrix K i and K j As well as the external parameter relationship, the equation group shown in formula 2.2 can be listed:

[0076]

[0077] Based on this, formula 2.2 can be solved to obtain the three-dimensional coordinates of the face area in the world coordinate system; if the three-dimensional coordinates obtained by triangulation are within a reasonable error range (consistent with other determined face positions or known position relationships in the scene), it is determined that the two face areas belong to the same face, so as to perform counting statistics and avoid repeated counting; illustratively, when the three-dimensional coordinates of any two face areas in the world coordinate system are located in a specified spatial range, it can be determined that any two face areas belong to the same face, and when the three-dimensional coordinates of any two face areas in the world coordinate system are not located in the specified spatial range, it can be determined that any two face areas do not belong to the same face, and so on. Optionally, the specified spatial range can be set according to experience or according to actual needs, and the embodiments of the present invention are not limited to this.

[0078] Optionally, in other embodiments, two face regions in a face region pair whose similarity is greater than a preset similarity threshold may be determined as the same face, and two face regions in a face region pair whose similarity is less than or equal to a preset similarity threshold may be determined as different faces, etc. Optionally, the preset similarity threshold may be set according to experience or actual needs, and the embodiments of the present invention are not limited thereto.

[0079] Based on this, the electronic device can determine whether any two faces in the images captured by each camera at any image capture time are the same (i.e., whether they are the same face) based on the face detection results of the images captured by each camera at any image capture time, so as to determine the number of different faces in the images captured by each camera at any image capture time, and use the number of different faces as the number of faces at any image capture time.

[0080] In another specific implementation, the electronic device may also determine the position (i.e., positioning) and posture (such as pitch angle, etc.) of each camera in space, such as through a calibration method; and based on the internal parameters (such as focal length, pixel size, etc.) and external parameters (such as positioning and posture, etc.) of each camera, determine the overlapping area of ​​the fields of view of n cameras, such as projecting the field of view range of each camera into three-dimensional space, thereby determining the overlapping part between them; illustratively, for two cameras with different viewing angles, their field of view cones in space can be obtained by calculating their optical axis directions and field of view angles, and the intersection of the two field of view cones is the overlapping area; based on this, based on the face detection results of the images collected by each camera at any image collection time, it is possible to judge in turn whether any two faces in the overlapping area of ​​the images collected by each camera at any image collection time are the same, thereby determining the number of different faces in the images collected by each camera at any image collection time, so as to use the number of different faces as the number of faces at any image collection time, and so on; the embodiments of the present invention are not limited to this.

[0081] Optionally, when determining the robot arm motion adjustment data of the robot arm based on the result of the population counting, the electronic device may use the result of the population counting to determine the amplitude adjustment factor, and determine the adjustment amplitude of the robot arm motion based on the amplitude adjustment factor and the original amplitude of the robot arm, such as the amplitude adjustment factor is B amp (H), H represents the result of the headcount, B amp (H) = 1 + 0.1H, then adjust the robot arm's movement range B = B0 × B amp (H); and / or, the number of people counted and the frequency adjustment factor can be used to determine the frequency of the robot arm movement, such as the frequency adjustment factor F freq (H), F freq (H) = 1 + 0.05H, then adjust the robot arm movement frequency F = F0 × F freq (H), etc.

[0082] Further, the mechanical arm motion parameter data of the mechanical arm can be determined based on the pending mechanical arm motion parameter data and the mechanical arm motion adjustment data. Optionally, the electronic device can add the pending mechanical arm motion speed to the mechanical arm motion parameter data; and / or, the pending mechanical arm motion amplitude and the adjusted mechanical arm motion amplitude can be weighted summed to obtain an amplitude integration result, when the amplitude integration result is less than or equal to a preset amplitude threshold, the amplitude integration result can be added to the mechanical arm motion parameter data, when the amplitude integration result is greater than the preset amplitude threshold, the preset amplitude threshold can be added to the mechanical arm motion parameter data; and / or, the pending mechanical arm motion frequency and the adjusted mechanical arm motion frequency can be weighted summed to obtain a frequency integration result, when the frequency integration result is less than or equal to the preset frequency threshold, the frequency integration result can be added to the mechanical arm motion parameter data, when the frequency integration result is greater than the preset frequency threshold, the preset frequency threshold can be added to the mechanical arm motion parameter data, so as to determine the mechanical arm motion parameter data of the mechanical arm, and so on.

[0083] In another implementation, the electronic device may also use the pending robot arm motion parameter data as the robot arm motion parameter data of the robot arm, and so on.

[0084] Optionally, the electronic device may generate a robot arm control instruction based on the target robot arm action and / or the robot arm action parameter data, and send the robot arm control instruction to the robot arm to control the robot arm to perform the target robot arm action. Optionally, in an embodiment of the present invention, the robot arm action parameter vector may be set as A (used to indicate the target robot arm action, etc.), and the robot arm action instruction vector may be set as I (i.e., the robot arm control instruction). Then, the robot arm action parameter data to the robot arm action instruction conversion model can be expressed as: I = G (A); wherein G () is a conversion function, which is defined according to the robot arm control instruction format.

[0085] Optionally, the electronic device may continuously iterate and execute steps S201-S205 to obtain the image acquisition data set in real time, thereby implementing the update of the target robot arm action, etc. Based on this, the embodiment of the present invention may adjust the mapping function L() (used to determine the target robot arm action) and the conversion function G() according to the feedback information after the robot arm performs the action (such as the action completion status (such as the completion percentage, etc.), the change of visual information (such as the change in the number of people and / or the change in the target object, etc.), so as to improve the robustness and accuracy of the algorithm; Optionally, the feedback information is set to l b , the adjusted mapping function is L′(), the adjusted conversion function is G′(), then the adjustment process can be expressed as: L′=L+ΔL(l b ), G′=G+ΔG(l b), which is used to represent the interactive feedback adjustment model, that is, the embodiment of the present invention can adjust the robot arm action according to the real-time environmental conditions; wherein, l b Indicates feedback information, including but not limited to at least one of the following: comp (completion status) and l vis (visual variation), etc., which is not limited in the embodiment of the present invention; ΔL and ΔG represent adjustment functions, which are used to correct the mapping function and the conversion function according to the feedback information.

[0086] After acquiring the image acquisition data set, the embodiment of the present invention can perform person detection on each image in the image acquisition data set to obtain the person detection data of each image, wherein the person detection data of one image includes the person segmentation mask image of the corresponding image and the person key point indication data of each person in the corresponding image, and the person segmentation mask image of one image includes the segmentation mask of each person in the corresponding image; and based on the person segmentation mask images of each image, the target object is determined. Accordingly, the target mechanical arm motion mapping indication data corresponding to the target object can be determined based on the person key point indication data of the target object. Based on this, the target mechanical arm motion can be determined based on the target mechanical arm motion mapping indication data, and the mechanical arm can be controlled to perform the target mechanical arm motion. It can be seen that the embodiment of the present invention can realize multi-dimensional visual information fusion interaction, that is, the embodiment of the present invention comprehensively considers multi-dimensional visual information such as the number of faces, key character postures, and object semantics, and applies this information fusion to the generation and control of the target robotic arm movement, thereby realizing a richer and more intelligent interaction effect (such as interactive dancing effect) between the robotic arm and the surrounding environment, breaking through the limitation of the single action setting of the traditional robotic arm, so that the robotic arm interaction method based on visual recognition can also be called a robotic arm interactive dance method based on visual recognition or a robotic arm interactive dance method based on visual recognition, etc.; and, the embodiment of the present invention can realize a dynamic switching mechanism of the target object, that is, it can dynamically switch when different target objects appear, so that the robotic arm can respond to the action changes of different characters in real time, thereby enhancing the flexibility and real-time nature of the interaction; in addition, the embodiment of the present invention can also realize action customization based on the semantics of key objects, that is, the action of the robotic arm can be customized according to the semantic characteristics of the object, so that the action of the robotic arm can be better matched with the objects in the environment, thereby improving the artistry and appreciation of the performance.

[0087] Based on the description of the above-mentioned embodiments of the robot arm interaction method based on visual recognition, the embodiment of the present invention further proposes a robot arm interaction device based on visual recognition, and the robot arm interaction device based on visual recognition can be a computer program (including program code) running in an electronic device; Figure 3As shown, the robot arm interactive device based on visual recognition may include an image acquisition module 301, an image analysis and processing module 302, an action generation module 303 and a robot arm control module 304. The robot arm interactive device based on visual recognition may perform Figure 1 or Figure 2 The robotic arm interaction method based on visual recognition shown, that is, the robotic arm interaction device based on visual recognition can run the above-mentioned units:

[0088] An image acquisition module 301 is used to acquire an image acquisition data set, wherein the image acquisition data set includes m image acquisition data, and one image acquisition data includes an image acquired by each camera of n cameras at an image acquisition time, and m and n are both positive integers;

[0089] The image analysis and processing module 302 is used to determine the target manipulator motion mapping indication data corresponding to the target object based on the image acquisition data set; wherein the target object is the object with the largest object area indication value among at least one object, and the object area indication value of an object is used to indicate the size of the area occupied by the corresponding object;

[0090] An action generation module 303 is used to determine a target manipulator action based on the target manipulator action mapping indication data;

[0091] The robot arm control module 304 is used to control the robot arm to perform the target robot arm action.

[0092] In one implementation, when the image analysis and processing module 302 determines the target mechanical arm motion mapping indication data corresponding to the target object based on the image acquisition data set, it can be specifically used to:

[0093] Performing person detection on each image in the image acquisition data set respectively to obtain person detection data of each image, wherein the person detection data of one image includes a person segmentation mask image of the corresponding image and person key point indication data of each person in the corresponding image, and the person segmentation mask image of one image includes a segmentation mask of each person in the corresponding image;

[0094] Determine the target object based on the person segmentation mask image of each image;

[0095] Based on the character key point indication data of the target object, the target mechanical arm motion mapping indication data corresponding to the target object is determined.

[0096] In another embodiment, when the image analysis and processing module 302 determines the target mechanical arm motion mapping indication data corresponding to the target object based on the key point indication data of the target object, it can be specifically used to:

[0097] Based on the key point indication data of the target object, determine the human body posture action of the target object, and add the human body posture action identifier of the human body posture action of the target object to the target mechanical arm action mapping indication data corresponding to the target object, so as to determine the target mechanical arm action; and / or,

[0098] Based on the character key point indication data of the target object, the action guidance data of the target object is determined, and the action guidance data is added to the target robotic arm action mapping indication data; wherein the action guidance data includes at least one of the following: character arm amplitude, character action speed and character action frequency, and the action guidance data supports the determination of robotic arm action parameter data of the robotic arm, so that the robotic arm performs the target robotic arm action according to the robotic arm action parameter data.

[0099] In another implementation, the image analysis and processing module 302 may also be used to:

[0100] Performing object detection on each image in the image acquisition data set to obtain object detection data of each image, wherein the object detection data of one image includes an object segmentation mask image of the corresponding image;

[0101] When determining the target object based on the person segmentation mask image of each image, the image analysis processing module 302 can be specifically used to:

[0102] Determine a target object based on the person segmentation mask image and the object segmentation mask image of each image, wherein the target object is a person or an object;

[0103] The image analysis processing module 302 may also be used for:

[0104] When the target object is a person, triggering execution of the person key point indication data based on the target object to determine the target mechanical arm motion mapping indication data corresponding to the target object;

[0105] When the target object is an article, the article semantic information of the target object is determined, and the article semantic information of the target object is added to the target robot arm action mapping indication data corresponding to the target object, and one article semantic information corresponds to one robot arm action.

[0106] In another embodiment, when the target object is a person, the target manipulator action mapping indication data includes a human body posture action identifier; when the target object is an object, the target manipulator action mapping indication data includes object semantic information; when the action generation module 303 determines the target manipulator action based on the target manipulator action mapping indication data, it can be specifically used to:

[0107] When the target mechanical arm motion mapping indication data includes a human body posture motion identifier, based on the human body posture motion identifier in the target mechanical arm motion mapping indication data, a target mechanical arm motion identifier matching the human body posture motion identifier in the target mechanical arm motion mapping indication data is determined from the first mechanical arm motion set, and the target mechanical arm motion identifier is used to indicate the target mechanical motion, so as to achieve the determination of the target mechanical arm motion;

[0108] When the target robotic arm motion mapping indication data includes object semantic information, based on the object semantic information in the target robotic arm motion mapping indication data, a target robotic arm motion identifier that matches the object semantic information in the target robotic arm motion mapping indication data is determined from the second robotic arm motion set.

[0109] In another embodiment, when the target object is a person, the target robot arm motion mapping indication data includes motion guidance data of the target object, and the motion guidance data includes at least one of the following: the amplitude of the person's arm, the speed of the person's motion, and the frequency of the person's motion; the motion generation module 303 can also be used to:

[0110] Based on the action guidance data, determine the pending mechanical arm action parameter data of the mechanical arm, the pending mechanical arm action parameter data including at least one of the following: a pending mechanical arm action amplitude, a pending mechanical arm action frequency, and a pending mechanical arm action speed; wherein the pending mechanical arm action amplitude is determined based on the character's arm amplitude, the pending mechanical arm action frequency is determined based on the character's action frequency, and the pending mechanical arm action speed is determined based on the character's action speed;

[0111] Determine the robot arm motion parameter data of the robot arm based on the pending robot arm motion parameter data;

[0112] When controlling the robotic arm to perform the target robotic arm action, the robotic arm control module 304 may be specifically used to:

[0113] According to the robot arm motion parameter data, the robot arm is controlled to perform the target robot arm motion.

[0114] In another implementation manner, when the action generation module 303 determines the manipulator action parameter data of the manipulator based on the pending manipulator action parameter data, it can be specifically used to:

[0115] Performing face detection on each image in the image acquisition data set to obtain a face detection result for each image;

[0116] Based on the face detection results of the respective images, a headcount result is determined, and based on the headcount result, a robot arm motion adjustment data of the robot arm is determined; the robot arm motion adjustment data includes an adjustment robot arm motion amplitude and / or an adjustment robot arm motion frequency of the robot arm;

[0117] Based on the pending robot arm motion parameter data and the robot arm motion adjustment data, the robot arm motion parameter data of the robot arm is determined.

[0118] According to one embodiment of the present invention, Figure 3 Each unit in the shown visual recognition-based robotic arm interactive device can be separately or completely combined into one or several other units to form a unit, or one (or some) of the units can be further divided into multiple functionally smaller units to form a unit, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present invention. The above-mentioned units are divided based on logical functions. In practical applications, the function of one unit can also be realized by multiple units, or the functions of multiple units can be realized by one unit. In other embodiments of the present invention, any visual recognition-based robotic arm interactive device can also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented by the collaboration of multiple units.

[0119] According to another embodiment of the present invention, the program can be executed by running a program on a general electronic device such as a computer including a central processing unit (CPU), a random access memory medium (RAM), a read-only memory medium (ROM), and other processing elements and storage elements. Figure 1 or Figure 2 A computer program (including program code) for each step involved in the corresponding method shown in Figure 3 The robot arm interaction device based on visual recognition shown in the embodiment of the present invention and the robot arm interaction method based on visual recognition are implemented. The computer program can be recorded on a computer storage medium, for example, and loaded into the above-mentioned electronic device through the computer storage medium and run therein.

[0120] Based on the description of the above method embodiment and device embodiment, the exemplary embodiment of the present invention further provides an electronic device, including: at least one processor; and a memory connected to the at least one processor in communication. The memory stores a computer program that can be executed by the at least one processor, and the computer program is used to enable the electronic device to perform the method according to the embodiment of the present invention when executed by the at least one processor.

[0121] Exemplary embodiments of the present invention also provide a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to perform a method according to an embodiment of the present invention.

[0122] An exemplary embodiment of the present invention further provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor of a computer, the computer is used to enable the computer to perform a method according to an embodiment of the present invention.

[0123] refer to Figure 4 , a block diagram of an electronic device 400 that can be used as a server or client of the present invention will now be described, which is an example of a hardware device that can be applied to various aspects of the present invention. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0124] like Figure 4 As shown, the electronic device 400 includes a computing unit 401, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 402 or a computer program loaded from a storage unit 408 into a random access memory (RAM) 403. In the RAM 403, various programs and data required for the operation of the electronic device 400 can also be stored. The computing unit 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0125] A plurality of components in the electronic device 400 are connected to the I / O interface 405, including: an input unit 406, an output unit 407, a storage unit 408, and a communication unit 409. The input unit 406 may be any type of device capable of inputting information to the electronic device 400, and the input unit 406 may receive input digital or character information, and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 407 may be any type of device capable of presenting information, and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 408 may include, but is not limited to, a disk, an optical disk. The communication unit 409 allows the electronic device 400 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks, and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0126] The computing unit 401 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 401 performs the various methods and processes described above. For example, in some embodiments, the mechanical arm interaction method based on visual recognition may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 408. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 400 via the ROM 402 and / or the communication unit 409. In some embodiments, the computing unit 401 may be configured to perform the mechanical arm interaction method based on visual recognition by any other appropriate means (e.g., by means of firmware).

[0127] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0128] In the context of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0129] As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0130] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0131] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0132] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0133] Furthermore, it should be understood that what is disclosed above is only a preferred embodiment of the present invention, and certainly cannot be used to limit the scope of rights of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.

Claims

1. A robotic arm interaction method based on visual recognition, characterized in that: include: Acquire an image acquisition data set, wherein the image acquisition data set includes m image acquisition data, one image acquisition data includes an image acquired by each camera of the n cameras at an image acquisition time, and m and n are both positive integers; Based on the image acquisition data set, determining the target manipulator motion mapping indication data corresponding to the target object; wherein the target object is an object with the largest object area indication value among at least one object, and the object area indication value of an object is used to indicate the size of the area occupied by the corresponding object; Based on the target robot arm motion mapping indication data, a target robot arm motion is determined, and the robot arm is controlled to perform the target robot arm motion.

2. The method according to claim 1, characterized in that The step of determining target mechanical arm motion mapping indication data corresponding to the target object based on the image acquisition data set includes: Performing person detection on each image in the image acquisition data set respectively to obtain person detection data of each image, wherein the person detection data of one image includes a person segmentation mask image of the corresponding image and person key point indication data of each person in the corresponding image, and the person segmentation mask image of one image includes a segmentation mask of each person in the corresponding image; Determine the target object based on the person segmentation mask image of each image; Based on the character key point indication data of the target object, the target mechanical arm motion mapping indication data corresponding to the target object is determined.

3. The method according to claim 2, characterized in that The step of determining target mechanical arm motion mapping indication data corresponding to the target object based on the key point indication data of the target object comprises: Based on the key point indication data of the target object, determine the human body posture action of the target object, and add the human body posture action identifier of the human body posture action of the target object to the target mechanical arm action mapping indication data corresponding to the target object, so as to determine the target mechanical arm action; and / or, Based on the character key point indication data of the target object, the action guidance data of the target object is determined, and the action guidance data is added to the target robotic arm action mapping indication data; wherein the action guidance data includes at least one of the following: character arm amplitude, character action speed and character action frequency, and the action guidance data supports the determination of robotic arm action parameter data of the robotic arm, so that the robotic arm performs the target robotic arm action according to the robotic arm action parameter data.

4. The method according to claim 2, characterized in that: The method further comprises: Performing object detection on each image in the image acquisition data set to obtain object detection data of each image, wherein the object detection data of one image includes an object segmentation mask image of the corresponding image; The step of determining the target object based on the person segmentation mask image of each image comprises: Determine a target object based on the person segmentation mask image and the object segmentation mask image of each image, wherein the target object is a person or an object; The method further comprises: When the target object is a person, triggering execution of the person key point indication data based on the target object to determine the target mechanical arm motion mapping indication data corresponding to the target object; When the target object is an article, the article semantic information of the target object is determined, and the article semantic information of the target object is added to the target robot arm action mapping indication data corresponding to the target object, and one article semantic information corresponds to one robot arm action.

5. The method according to any one of claims 1 to 4, characterized in that: When the target object is a person, the target manipulator action mapping indication data includes a human body posture action identifier; when the target object is an object, the target manipulator action mapping indication data includes object semantic information; and determining the target manipulator action based on the target manipulator action mapping indication data includes: When the target mechanical arm motion mapping indication data includes a human body posture motion identifier, based on the human body posture motion identifier in the target mechanical arm motion mapping indication data, a target mechanical arm motion identifier matching the human body posture motion identifier in the target mechanical arm motion mapping indication data is determined from the first mechanical arm motion set, and the target mechanical arm motion identifier is used to indicate the target mechanical motion, so as to achieve the determination of the target mechanical arm motion; When the target robotic arm motion mapping indication data includes object semantic information, based on the object semantic information in the target robotic arm motion mapping indication data, a target robotic arm motion identifier that matches the object semantic information in the target robotic arm motion mapping indication data is determined from the second robotic arm motion set.

6. The method according to any one of claims 1 to 4, characterized in that: When the target object is a person, the target robot arm motion mapping indication data includes motion guidance data of the target object, and the motion guidance data includes at least one of the following: the amplitude of the person's arm, the speed of the person's motion, and the frequency of the person's motion; the method further includes: Based on the action guidance data, determine the pending mechanical arm action parameter data of the mechanical arm, the pending mechanical arm action parameter data including at least one of the following: a pending mechanical arm action amplitude, a pending mechanical arm action frequency, and a pending mechanical arm action speed; wherein the pending mechanical arm action amplitude is determined based on the character's arm amplitude, the pending mechanical arm action frequency is determined based on the character's action frequency, and the pending mechanical arm action speed is determined based on the character's action speed; Determine the robot arm motion parameter data of the robot arm based on the pending robot arm motion parameter data; The controlling the mechanical arm to perform the target mechanical arm action comprises: According to the robot arm motion parameter data, the robot arm is controlled to perform the target robot arm motion.

7. The method according to claim 6, characterized in that The step of determining the robot arm motion parameter data of the robot arm based on the pending robot arm motion parameter data comprises: Performing face detection on each image in the image acquisition data set to obtain a face detection result for each image; Based on the face detection results of the respective images, a headcount result is determined, and based on the headcount result, a robot arm motion adjustment data of the robot arm is determined; the robot arm motion adjustment data includes an adjustment robot arm motion amplitude and / or an adjustment robot arm motion frequency of the robot arm; Based on the pending robot arm motion parameter data and the robot arm motion adjustment data, the robot arm motion parameter data of the robot arm is determined.

8. A robotic arm interactive device based on visual recognition, characterized in that: The device comprises: An image acquisition module is used to acquire an image acquisition data set, wherein the image acquisition data set includes m image acquisition data, one image acquisition data includes an image acquired by each camera of n cameras at an image acquisition time, and m and n are both positive integers; An image analysis and processing module, used to determine the target manipulator motion mapping indication data corresponding to the target object based on the image acquisition data set; wherein the target object is an object with the largest object area indication value among at least one object, and the object area indication value of an object is used to indicate the size of the area occupied by the corresponding object; An action generation module, used for determining a target manipulator action based on the target manipulator action mapping indication data; The robot arm control module is used to control the robot arm to perform the target robot arm action.

9. An electronic device, characterized in that: include: processor; and a memory for storing programs, The program includes instructions, which, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to make a computer execute the method according to any one of claims 1-7.

Images