Interaction task state determination method and system and electronic equipment

Abstract

The invention relates to the technical field of behavior analysis, and discloses an interaction task state determination method and system and electronic equipment. According to the method, an interaction task is divided into interaction operation nodes corresponding to interaction behaviors or independent behaviors, operation images corresponding to the interaction operation nodes are obtained, node states corresponding to the interaction behaviors are determined according to the area intersection-to-union ratio of operation objects between image areas, object feature vectors are determined according to key points of the operation objects, and the object feature vectors are determined according to the key points of the operation objects. Compared with setting a visual algorithm by taking the interaction task as a whole interaction behavior, the interaction task is divided into interaction operation nodes corresponding to the interaction behavior or the independent behavior according to the interaction type, and the node state of each interaction operation node is judged; when the node state is judged, the specific behavior or object state is focused according to the interaction type, whether the interaction task is completed or not is accurately judged according to all the node states of the interaction task, and the accuracy of task state monitoring is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of behavior analysis, and in particular to a method, system and electronic device for determining an interactive task state. Background Art

[0002] In the automotive manufacturing industry, a field that pursues ultimate automation and precision, the monitoring and management of production processes are directly related to the stability of product quality, the improvement of production efficiency and the optimization of overall operational benefits. However, traditional manual monitoring methods, due to their high costs, inherent error-proneness and difficult-to-overcome low efficiency problems, have gradually failed to meet the urgent needs of modern automobile production lines for efficient and accurate monitoring. Using advanced computer vision technology to replace manual monitoring and realize automated and intelligent monitoring of production processes has become an important driving force for promoting the transformation and upgrading of the industry.

[0003] Among them, computer vision algorithms have been widely used in production process monitoring due to their powerful image processing capabilities, especially in spatial position recognition and state judgment of interactive tasks between objects. By capturing images of parts or tools on the production line in real time and combining advanced image processing, feature extraction and pattern recognition technologies, these algorithms can accurately analyze and judge interactive tasks in the production process.

[0004] However, complex interactive tasks are often not limited to direct physical contact or position changes, but also include a large number of independent behaviors, such as waiting for workpieces, pausing transportation, and independent movement during transportation. These independent behaviors lack obvious spatial position change characteristics, making it difficult for existing monitoring algorithms based on spatial position judgment to accurately capture and analyze the indirect interactive information. As a result, in determining whether such interactive tasks are completed, existing detection algorithms show a low accuracy rate and cannot provide reliable guidance for production. Summary of the invention

[0005] In order to provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. The summary is not an extensive review, nor is it intended to identify key / critical components or delineate the scope of protection of these embodiments, but rather serves as a prelude to the detailed description that follows.

[0006] In view of the above-mentioned shortcomings of the prior art, the present application discloses a method, system and electronic device for determining an interactive task state, so as to improve the accuracy of determining the interactive task state.

[0007] The present application discloses a method for determining an interactive task state, comprising: obtaining a job image corresponding to an interactive job node, wherein the interactive job node is obtained by splitting the interactive task according to the interaction type, and the job image includes image areas corresponding to a plurality of job objects respectively; if the interaction type corresponding to the interactive job node includes an interactive behavior, then obtaining the area areas corresponding to each of the image areas respectively, and calculating the area intersection and union ratio between the area areas of each of the area areas, so as to determine the node state corresponding to the interactive job node according to the area intersection and union ratio; if the interaction type corresponding to the interactive job node includes an independent behavior, then performing key point recognition on the image area corresponding to the target object to obtain the object key points, and performing feature extraction based on the key point coordinates corresponding to the object key points to obtain the object feature vector, so as to determine the node state corresponding to the interactive job node according to the object feature vector, wherein the target object includes at least a part of the job object; and determining the task state corresponding to the interactive task according to the node state corresponding to each interactive job node in the interactive task.

[0008] In one embodiment of the present application, the operation object includes at least a part of a human object, a workpiece object, and a workbench object, and the method also includes at least one of the following: if the node type includes interactive behavior, and the image area corresponding to the workpiece object is outside the image area corresponding to the workbench object, then the node state corresponding to the interactive operation node is determined to be an unfinished state, wherein the workbench object is used to carry the workpiece object; if the node type includes interactive behavior, and if the object key point corresponding to the human object is outside the image area corresponding to the workpiece object, then the node state corresponding to the interactive operation node is determined to be an unfinished state; if the object category corresponding to the workpiece object does not belong to a preset category, then the node state corresponding to the interactive operation node is determined to be an unfinished state, wherein the object category is obtained by object recognition of the workpiece object in the operation image.

[0009] In one embodiment of the present application, key point recognition is performed on an image area corresponding to a target object to obtain object key points, including: if the target object is a human object, multiple first human images are obtained; human key points in the first human image are marked to obtain sample labels, wherein the human key points include at least one of an elbow area, a wrist area, and a shoulder area; an occluded area where human key points exist is determined from the first human image, and at least a part of the occluded area in the first human image is replaced with a preset image to be replaced to obtain a second human image, wherein no human key points exist in the image to be replaced; the first human image and / or the second human image are used as model training samples, and model training is performed according to the model training samples with sample labels to obtain a posture recognition model, wherein the posture recognition model is used to perform key point recognition on the image area corresponding to the target object to obtain object key points.

[0010] In one embodiment of the present application, model training is performed according to model training samples with sample labels to obtain a posture recognition model, including: obtaining a basic training model, wherein the basic training model includes a SCANet network model, and the SCANet network model is composed of a SCAneck module, a SCAblock module, a Sandglass module and a CoordAttention module; generating a first loss function according to a first real heat map and a first predicted heat map, wherein human body key points are labeled according to each of the model training samples to obtain a first real heat map corresponding to each human body key point, and target detection is performed on the model training samples based on the human body key points through the SCANet network model to obtain a first predicted heat map corresponding to each human body key point; obtaining an association relationship between each of the human body key points, and annotating each of the human body key points according to the association relationship The system is divided into multiple key point groups, wherein two human key points in the same key point group and having an associated relationship are mutually associated key points; a second loss function is generated according to a second real heat map and a second predicted heat map, wherein the human key points and the associated key points corresponding to the human key points are annotated according to each of the model training samples to obtain the second real heat maps corresponding to each of the human key points, and the model training samples are subjected to target detection based on the human key points and the associated key points corresponding to the human key points through the SCANet network model to obtain the second predicted heat maps corresponding to each of the human key points; a model loss function corresponding to the SCANet network model is generated according to the first loss function and the second loss function, and the SCANet network model is trained according to the model training samples with sample labels, and the trained SCANet network model is used as a posture recognition model.

[0011] In one embodiment of the present application, the posture recognition model performs key point recognition in the following manner: inputting the image area corresponding to the target object into the posture recognition model as a feature map; adjusting the number of channels and / or spatial resolution of the feature map through multiple SCAneck modules to extract features from the feature map; extracting features from the feature map through multiple SCAblock modules, wherein each of the SCAblock modules corresponds to a different resolution and each of the SCAblock modules corresponds to a different number of channels; adjusting the reduction layer position and the expansion layer position in the posture recognition model through the Sandglass module; and outputting the feature map with attention weights through the CoordAttention module.

[0012] In one embodiment of the present application, feature extraction is performed based on the key point coordinates corresponding to the object key points to obtain an object feature vector, including: calculating the distance between the object key points with an associated relationship based on the key point coordinates corresponding to each of the object key points to obtain an associated distance vector; establishing key point lines based on the associated relationship between the object key points, and calculating based on the key point coordinates corresponding to each of the object key points to obtain an associated angle vector between the key point lines; generating an object feature vector based on the associated distance vector and / or the associated angle vector.

[0013] In one embodiment of the present application, obtaining a job image corresponding to the interactive job node includes: in response to executing the interactive job node, performing image acquisition on each of the job objects to obtain a job image stream, wherein the job image stream includes multiple image frames; determining any image frame as a job image, performing target detection on each of the job objects in the job image, and using the ROI area obtained by the target detection as the image area corresponding to the job object.

[0014] In one embodiment of the present application, the method also includes: obtaining a current job flow, wherein the current job flow is composed of a plurality of interactive tasks arranged in a preset execution order; splitting each of the interactive tasks into a task start node and a task end node, and using the task start node and the task end node as an interactive job node respectively; determining the interactive job node of the first order as a target node according to the preset execution order; obtaining a node status corresponding to the target node in response to the target node; if the node status corresponding to the target node is a completed state, determining the interactive job node of the next order after the target node as a new target node according to the preset execution order; and generating a process execution status corresponding to the current job flow according to the node status corresponding to each of the interactive job nodes.

[0015] The present application discloses an interactive task state determination system, comprising: an acquisition module, used to acquire a job image corresponding to an interactive job node, wherein the interactive job node is obtained by splitting the interactive task according to the interaction type, and the job image includes image areas corresponding to multiple job objects respectively; a first determination module, used to acquire the area areas corresponding to each of the image areas respectively if the interaction type corresponding to the interactive job node includes interactive behavior, and calculate the area intersection and union ratio between the area areas of each of the area areas, so as to determine the node state corresponding to the interactive job node according to the area intersection and union ratio; a second determination module, used to perform key point recognition on the image area corresponding to the target object to obtain the object key points if the interaction type corresponding to the interactive job node includes independent behavior, and perform feature extraction according to the key point coordinates corresponding to the object key points to obtain the object feature vector, so as to determine the node state corresponding to the interactive job node according to the object feature vector, wherein the target object includes at least a part of the job object; a third determination module, used to determine the task state corresponding to the interactive task according to the node states corresponding to each interactive job node in the interactive task.

[0016] The present application discloses an electronic device, comprising: a processor and a memory; the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the electronic device executes the above method.

[0017] Beneficial effects of this application:

[0018] By splitting the interactive task into interactive operation nodes corresponding to interactive behaviors or independent behaviors, and obtaining the operation images corresponding to the interactive operation nodes, the node state corresponding to the interactive behavior is determined according to the area intersection and union ratio between the operation objects in the image regions, and the object feature vector is determined according to the key points of the operation object, so as to determine the node state corresponding to the independent behavior according to the object feature vector. In this way, compared with setting the visual algorithm for the interactive task as a whole interactive behavior, the interactive task is divided into interactive operation nodes corresponding to interactive behaviors or independent behaviors according to the interaction type, and the node state of each interactive operation node is judged respectively. When judging the node state, the focus is placed on the specific behavior or object state according to the interaction type, making the state judgment more detailed and accurate, and then accurately judging whether the interactive task is completed according to all the node states of the interactive task, thereby improving the accuracy of task state monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a flowchart of a method for determining an interactive task state in an embodiment of the present application;

[0020] Figure 2 It is a structural diagram of a platform architecture for implementing an interactive task state determination method in an embodiment of the present application;

[0021] Figure 3 is a flowchart of another interactive task status determination method in an embodiment of the present application;

[0022] Figure 4 It is a flowchart of a method for determining a node state of an interactive behavior in an embodiment of the present application;

[0023] Figure 5 It is a flowchart of a method for determining node status of independent behavior in an embodiment of the present application;

[0024] Figure 6 It is a flowchart of a training method of a posture recognition model in an embodiment of the present application;

[0025] Figure 7 It is a schematic diagram of the structure of a key point of a human body in an embodiment of the present application;

[0026] Figure 8 It is a structural diagram of a SCANet network model in an embodiment of the present application;

[0027] Fig. 9 It is a schematic diagram of the structure of a SCAneck module and a SCAblock module in an embodiment of the present application;

[0028] Fig.10 is a schematic diagram of the structure of a Sandglass module in an embodiment of the present application;

[0029] Fig.11 It is a schematic diagram of the structure of a CoordAttention module in an embodiment of the present application;

[0030] Fig.12 is a structural diagram of an interactive task state determination system in an embodiment of the present invention;

[0031] Fig.13 It is a schematic diagram of the structure of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The following describes the embodiments of the present invention through specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and sub-samples in the embodiments can be combined with each other without conflict.

[0033] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0034] In the following description, numerous details are discussed to provide a more thorough explanation of the embodiments of the present invention. However, it is obvious to those skilled in the art that the embodiments of the present invention can be implemented without these specific details. In other embodiments, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid making the embodiments of the present invention difficult to understand.

[0035] The terms "first", "second", etc. in the specification and claims of the embodiments of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged where appropriate, so that the embodiments of the embodiments of the present disclosure described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions.

[0036] Unless otherwise stated, the term "plurality" means two or more.

[0037] In the embodiment of the present disclosure, the character " / " indicates that the preceding and following objects are in an "or" relationship. For example, A / B indicates: A or B.

[0038] The term "and / or" is a description of the association relationship between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or, A and B.

[0039] Combination Figure 1 As shown, the embodiment of the present disclosure provides a method for determining an interactive task state, including:

[0040] Step S101, obtaining a job image corresponding to an interactive job node;

[0041] Among them, the interactive job nodes are obtained by splitting the interactive tasks according to the interaction types;

[0042] Wherein, the operation image includes image areas corresponding to a plurality of operation objects respectively;

[0043] Step S102, if the interaction type corresponding to the interactive operation node includes interactive behavior, then obtaining the area corresponding to each image area, and calculating the area intersection-and-union ratio between the areas of each area, so as to determine the node state corresponding to the interactive operation node according to the area intersection-and-union ratio;

[0044] Step S103: if the interaction type corresponding to the interactive operation node includes independent behavior, key point recognition is performed on the image area corresponding to the target object to obtain object key points, and feature extraction is performed according to the key point coordinates corresponding to the object key points to obtain an object feature vector, so as to determine the node state corresponding to the interactive operation node according to the object feature vector;

[0045] Wherein, the target object includes at least a part of the operation object;

[0046] Step S104: determining the task state corresponding to the interactive task according to the node state corresponding to each interactive operation node in the interactive task.

[0047] The interactive task status determination method provided by the embodiment of the present disclosure is adopted, by splitting the interactive task into interactive operation nodes corresponding to interactive behaviors or independent behaviors, and obtaining the operation image corresponding to the interactive operation node, the node status corresponding to the interactive behavior is determined according to the area intersection and union ratio between the operation objects in the image area, and the object feature vector is determined according to the key points of the operation object, so as to determine the node status corresponding to the independent behavior according to the object feature vector. In this way, compared with setting the visual algorithm for the interactive task as a whole interactive behavior, the interactive task is divided into interactive operation nodes corresponding to the interactive behavior or independent behavior according to the interaction type, and the node status of each interactive operation node is judged respectively, and then the node status is judged according to the interaction type to focus on the specific behavior or object status, so that the status judgment is more detailed and accurate, and then the completion of the interactive task is accurately judged according to all the node status of the interactive task, thereby improving the accuracy of task status monitoring.

[0048] Optionally, the working object includes at least a part of a human body object, a workpiece object, and a workbench object.

[0049] Combination Figure 2 As shown, the embodiment of the present disclosure provides a platform architecture for implementing an interactive task state determination method, including a video acquisition module 201, a target detection module 202, a posture estimation module 203, a node state module 204, and a task state module 205.

[0050] The video acquisition module 201 is used to obtain the operation image corresponding to the interactive operation node.

[0051] The target detection module 202 is used to perform target detection on each work object in the work image to obtain the image area corresponding to each work object and the workpiece type corresponding to the workpiece object, wherein the ROI area corresponding to the human object is sent to the posture estimation module 203, and the ROI area corresponding to the workpiece object and the workpiece type corresponding to the workpiece object are sent to the node status module 204.

[0052] The posture estimation module 203 is used to perform key point recognition on the image area corresponding to the target object, obtain the object key points, and send the object key points to the node posture module 204.

[0053] The node status module 204 is used to obtain the area corresponding to each image area respectively, and calculate the area intersection and union ratio between the areas of each area, if the interaction type corresponding to the interactive operation node includes interactive behavior, so as to determine the node state corresponding to the interactive operation node according to the area intersection and union ratio; if the interaction type corresponding to the interactive operation node includes independent behavior, feature extraction is performed according to the key point coordinates corresponding to the object key points to obtain the object feature vector, so as to determine the node state corresponding to the interactive operation node according to the object feature vector.

[0054] The task status module 205 determines the task status of the interactive task according to the node status corresponding to each interactive operation node.

[0055] The platform framework for implementing the interactive task status determination method provided by the embodiment of the present disclosure is adopted. By splitting the interactive task into interactive operation nodes corresponding to interactive behaviors or independent behaviors, and obtaining the operation image corresponding to the interactive operation node, the node state corresponding to the interactive behavior is determined according to the area intersection and union ratio between the operation objects in the image regions, and the object feature vector is determined according to the key points of the operation object, so as to determine the node state corresponding to the independent behavior according to the object feature vector. In this way, compared with setting the visual algorithm for the interactive task as a whole interactive behavior, the interactive task is divided into interactive operation nodes corresponding to the interactive behavior or independent behavior according to the interaction type, and the node state of each interactive operation node is judged respectively. When judging the node state, the focus is placed on the specific behavior or object state according to the interaction type, so that the state judgment is more detailed and accurate, and then the completion of the interactive task is accurately judged according to all the node states of the interactive task, thereby improving the accuracy of task state monitoring.

[0056] Optionally, obtaining a job image corresponding to an interactive job node includes: in response to executing the interactive job node, performing image acquisition on each job object to obtain a job image stream, wherein the job image stream includes multiple image frames; determining any image frame as a job image, performing target detection on each job object in the job image, and using the ROI area obtained by the target detection as the image area corresponding to the job object.

[0057] In some embodiments, the interactive tasks of the operators are captured by video through a video capture device to obtain an operation image stream; the video capture device has a highly advanced autofocus function, which can automatically adjust according to the distance between the operator's work content and the camera to obtain a clear and complete picture; at the same time, the video capture device also has an automatic fill light function, which can effectively fill light in a low-light environment, thereby accurately capturing and recording the operator's work behavior.

[0058] In some embodiments, in the collected job image stream, the image frames are not continuous, but are sampled by interval frames, which can reduce the amount of processed data, improve the processing speed, and ensure that the acquired video stream information is rich enough.

[0059] In some embodiments, the recognition task of each work object is integrated into a target detection module, and target detection is performed on each work object in the work image based on the YOLO algorithm to obtain the image area and object type corresponding to each work object, thereby avoiding network communication redundancy and repeated calculations, and greatly optimizing the node status judgment market.

[0060] Optionally, the method also includes: obtaining a current job flow, wherein the current job flow is composed of a plurality of interactive tasks arranged in a preset execution order; splitting each interactive task into a task start node and a task end node, and using the task start node and the task end node as an interactive job node respectively; determining the interactive job node with the first priority as a target node according to the preset execution order; obtaining a node status corresponding to the target node in response to the target node; if the node status corresponding to the target node is a completed status, determining the interactive job node with the next priority after the target node as a new target node according to the preset execution order; and generating a process execution status corresponding to the current job flow according to the node status corresponding to each interactive job node.

[0061] In some embodiments, the current operation process includes installing a car display screen, wherein the six interactive tasks in the current operation process are, in sequence, taking the display screen, installing the cover plate, installing the cable, installing the PCB board, inserting the cable, and scanning the code for release; a task start node and a task end node are defined for each interactive task, and 12 interactive operation nodes arranged in sequence are obtained.

[0062] Optionally, the method also includes: if the node type includes interactive behavior, and the image area corresponding to the workpiece object is outside the image area corresponding to the workbench object, then the node state corresponding to the interactive operation node is determined as an unfinished state, wherein the workbench object is used to carry the workpiece object.

[0063] Optionally, the method further includes: if the node type includes interactive behavior, and if the object key point corresponding to the human object is located outside the image area corresponding to the workpiece object, then determining the node state corresponding to the interactive operation node as an unfinished state.

[0064] Optionally, the method further includes: if the object category corresponding to the workpiece object does not belong to a preset category, determining the node state corresponding to the interactive operation node as an unfinished state, wherein the object category is obtained by performing object recognition on the workpiece object in the operation image.

[0065] Combination Figure 3 As shown, the embodiment of the present disclosure provides a method for determining an interactive task state, including:

[0066] Step S301, determining a target node from the interactive operation nodes corresponding to an interactive task;

[0067] Step S302, determine whether the target node is an interactive behavior, if so, jump to step S303, if not, jump to step S304;

[0068] Step S303, obtaining the area corresponding to each image area, and calculating the area intersection-and-union ratio between the area of ​​each area, so as to determine the node state corresponding to the interactive operation node according to the area intersection-and-union ratio, and then jump to step S305;

[0069] Step S304, performing key point recognition on the image area corresponding to the target object to obtain the object key points, and performing feature extraction according to the key point coordinates corresponding to the object key points to obtain the object feature vector, so as to determine the node state corresponding to the interactive operation node according to the object feature vector, and then jump to step S305;

[0070] Step S305: determining the task status of the interactive task according to the node status corresponding to each interactive operation node in the interactive task.

[0071] The platform framework for implementing the interactive task status determination method provided by the embodiment of the present disclosure is adopted. By splitting the interactive task into interactive operation nodes corresponding to interactive behaviors or independent behaviors, and obtaining the operation image corresponding to the interactive operation node, the node state corresponding to the interactive behavior is determined according to the area intersection and union ratio between the operation objects in the image regions, and the object feature vector is determined according to the key points of the operation object, so as to determine the node state corresponding to the independent behavior according to the object feature vector. In this way, compared with setting the visual algorithm for the interactive task as a whole interactive behavior, the interactive task is divided into interactive operation nodes corresponding to the interactive behavior or independent behavior according to the interaction type, and the node state of each interactive operation node is judged respectively. When judging the node state, the focus is placed on the specific behavior or object state according to the interaction type, so that the state judgment is more detailed and accurate, and then the completion of the interactive task is accurately judged according to all the node states of the interactive task, thereby improving the accuracy of task state monitoring.

[0072] In some embodiments, the intersection over union (IoU) is the ratio of two target regions between the region intersection and the region union, and is often used to evaluate the degree of overlap between two target regions.

[0073] Combination Figure 4 As shown, the embodiment of the present disclosure provides a method for determining a node state for interactive behavior, including:

[0074] Step S401, obtaining an image region corresponding to a human body object, an image region corresponding to a workpiece object, an image region corresponding to a workbench object, and an object category corresponding to the workpiece object;

[0075] Step S402, determining whether the image area corresponding to the workpiece object is located at the image coordinates corresponding to the workbench object, if yes, jump to step S403, if no, jump to step S402;

[0076] Step S403, determining whether the object key point corresponding to the human body object is located at the image coordinate corresponding to the workpiece object, if so, jump to step S404, if not, jump to step S402;

[0077] The object key points corresponding to the human body object are obtained by performing key point recognition on the image area corresponding to the human body object;

[0078] Step S404, calculating the area intersection-to-union ratio according to the image area corresponding to the human body object and the image area corresponding to the workpiece object;

[0079] Step S405, determining whether the area intersection-over-union ratio meets a preset intersection-over-union ratio threshold, if yes, jump to step S406, if no, jump to step S402;

[0080] Wherein, the preset intersection-and-union ratio threshold is satisfied and at least one of the following three situations exists: greater than the intersection-and-union ratio threshold, equal to the intersection-and-union ratio threshold, and less than the intersection-and-union ratio threshold;

[0081] Step S406: determine the node state corresponding to the target node as node completion.

[0082] In some embodiments, if the interactive task is to pick up the display screen, and the target node is the task end node of picking up the display screen, then at the beginning of the target node, there is an interactive behavior between the human object and the car display screen; first, determine whether the car display screen is located in the image area of ​​the workbench object; then, determine whether the human body key points of the human object are in the ROI area of ​​the car display screen; finally, calculate the area intersection and union ratio based on the ROI area of ​​the human object and the ROI area of ​​the car display screen; if the area intersection and union ratio is greater than a preset intersection and union ratio threshold, it is determined that the human object has picked up the car display screen.

[0083] In some embodiments, based on different requirements of interactive operations, some of the key points of the human body can be extracted as key points for determining whether the target node is completed. For example, in the task of installing a car display screen, interaction between the hands and the display screen is required. Therefore, the area intersection and union ratio between the hand ROI area and the display screen ROI area is calculated, and the node state of the target node is determined based on the calculation result, thereby reducing the number of reference targets and avoiding determining the node state of irrelevant ROI areas, thereby improving the success rate of determining the node state.

[0084] In some embodiments, the area intersection-over-union ratio is calculated by the following formula:

[0085]

[0086] In formula (1), IoU is the intersection over union ratio between image region A and image region B, S A is the area of ​​the image region A, S B is the area of ​​image region B.

[0087] Combination Figure 5 As shown, the embodiment of the present disclosure provides a node state determination method for independent behavior, including:

[0088] Step S501, obtaining the key point coordinates corresponding to each object key point;

[0089] Step S502, calculating the distance between the key points of the objects having the association relationship according to the key point coordinates corresponding to the key points of each object, and obtaining the association distance vector;

[0090] Step S503, establishing key point lines according to the association relationship between the key points of each object, and calculating according to the key point coordinates corresponding to the key points of each object, to obtain the association angle vector between the key point lines;

[0091] Step S504, generating an object feature vector according to the associated distance vector and / or the associated angle vector;

[0092] Step S505, performing similarity calculation between the preset feature vector and the calculated object feature vector;

[0093] Step S506: If the calculated feature similarity is greater than a preset similarity threshold, the node state corresponding to the target node is determined as node completion.

[0094] Optionally, feature extraction is performed based on key point coordinates corresponding to the object key points to obtain an object feature vector, including: calculating the distance between object key points with an associated relationship based on the key point coordinates corresponding to each object key point to obtain an associated distance vector; establishing key point lines based on the associated relationship between each object key point, and calculating based on the key point coordinates corresponding to each object key point to obtain an associated angle vector between each key point line; generating an object feature vector based on the associated distance vector and / or the associated angle vector.

[0095] In some embodiments, in addition to extracting key points from human objects, key points can also be identified for certain workpieces to obtain key points. For example, a fixed point in a movable part can be used as a key point to determine whether the movable workpiece is assembled based on the angle between the key point lines.

[0096] In some embodiments, if the interactive task is to pick up a display screen, and the target node is the task start node of picking up the display screen, then at the beginning of the target node, the human object and the target object (car display screen) are independent behaviors, and there is no interaction between the two; calculations are performed based on the key points of the human object such as the wrist, elbow, and shoulder to obtain an object feature vector, wherein the key point feature vector includes an associated distance vector and an associated angle vector; similarity is calculated between a preset feature vector and the calculated object feature vector; if the calculated feature similarity is greater than a preset similarity threshold, it is determined that the task start node of picking up the display screen has been completed.

[0097] In some embodiments, feature similarity is calculated using the following formula:

[0098]

[0099] In formula (2), cos(a, b) is the feature similarity between feature vector a and feature vector b, a·b is the dot product between feature vector a and feature vector b, ||a|| is the Euclidean length (norm) of feature vector a, and ||b|| is the Euclidean length of feature vector b.

[0100] Combination Figure 6 As shown, the embodiment of the present disclosure provides a training method for a posture recognition model, including:

[0101] Step S601, obtaining a plurality of first human body images and a sample label corresponding to each first human body image;

[0102] Among them, the sample labels include key points of the human body;

[0103] Step S602, determining an occluded area where key points of the human body are present in the first human body image, and replacing at least a portion of the occluded area in the first human body image with a preset image to be replaced, to obtain a second human body image;

[0104] Among them, there are no human key points in the image to be replaced;

[0105] Step S603, obtaining the SCANet network model;

[0106] Among them, the SCANet network model consists of SCAneck module, SCAblock module, Sandglass module and CoordAttention module;

[0107] Step S604, dividing the human body key points into a plurality of key point groups according to the association relationship between the human body key points;

[0108] Step S605, establishing a first loss function according to the key points of the human body, and establishing a second loss function according to the association relationship between the key points of the human body, so as to generate a model loss function corresponding to the SCANet network model according to the first loss function and the second loss function;

[0109] Step S606: taking the first human body image and / or the second human body image as a model training sample, and dividing the model training sample into a training set, a validation set and a test set;

[0110] Step S607, training the SCANet network model according to the divided sample set and the model loss function to obtain a posture recognition model.

[0111] Optionally, key point recognition is performed on the image area corresponding to the target object to obtain object key points, including: if the target object is a human object, multiple first human images are obtained; human key points in the first human image are marked to obtain sample labels, wherein the human key points include at least one of the elbow area, the wrist area and the shoulder area; the occluded area where the human key points exist is determined from the first human image, and at least a part of the occluded area in the first human image is replaced with a preset image to be replaced to obtain a second human image, wherein the human key points do not exist in the image to be replaced; the first human image and / or the second human image are used as model training samples, and model training is performed according to the model training samples with sample labels to obtain a posture recognition model, wherein the posture recognition model is used to perform key point recognition on the image area corresponding to the target object to obtain object key points.

[0112] In some embodiments, the ROI area corresponding to the human body object is input into the posture recognition model, and the posture recognition model calculates the pixel coordinates of the key points of the operator's body. The posture recognition model is a top-down human posture estimation algorithm, which better solves the problem of human body overlap caused by multiple people interfering with the operator.

[0113] Combination Figure 7 As shown, the embodiment of the present disclosure obtains multiple human key points by dividing the human body object, including the head area, neck area, left shoulder area, left elbow area, left wrist area, right shoulder area, right elbow area, right wrist area, left hip area, left knee area, left foot area, right hip area, right knee area, and right foot area.

[0114] In some embodiments, a detection frame corresponding to a human object is obtained by performing human body detection on the first human body image; since there are a large number of hand-object interactions in the automobile display screen installation task, an occlusion area is generated with the human body key point as the center, and the width and height of the occlusion area are adjusted according to the size of the detection frame corresponding to the human body object, and the calculation formula is as follows:

[0115] h=α·H1,w=β·W1 Formula (3)

[0116] In formula (3), h is the height of the image to be replaced, α is the preset image height ratio, H1 is the height of the detection frame, w is the width of the image to be replaced, β is the preset image width ratio, and W1 is the width of the detection frame; if the coordinates corresponding to the human key points are (x, y), based on formula (1), the coordinates of the upper left corner vertex of the occluded area are (xw / 2, yh / 2), and the coordinates of the lower right corner vertex of the occluded area are (x+w / 2, y+h / 2); by coloring the objects other than the human body object in the first human image, the obtained color is used as the color value of the image to be replaced, and the occluded area is replaced with the image to be replaced.

[0117] In this way, compared with directly training the posture recognition model through human body images, the data enhancement strategy of occluding the human body key points in the sample helps the posture recognition model predict the human body key points under occlusion.

[0118] Optionally, model training is performed according to model training samples with sample labels to obtain a posture recognition model, including: obtaining a basic training model, wherein the basic training model includes a SCANet network model, and the SCANet network model is composed of a SCAneck module, a SCAblock module, a Sandglass module and a CoordAttention module; generating a first loss function according to a first real heat map and a first predicted heat map, wherein human body key points are labeled according to each model training sample to obtain a first real heat map corresponding to each human body key point, and target detection is performed on the model training samples based on the human body key points through the SCANet network model to obtain a first predicted heat map corresponding to each human body key point; obtaining a correlation relationship between each human body key point, and dividing each human body key point according to the correlation relationship. The method is divided into a plurality of key point groups, wherein two human key points in the same key point group and having an associated relationship are mutually associated key points; a second loss function is generated according to the second real heat map and the second predicted heat map, wherein the human key points and the associated key points corresponding to the human key points are annotated according to each model training sample, and the second real heat map corresponding to each human key point is obtained, and the model training samples are subjected to target detection based on the human key points and the associated key points corresponding to the human key points through the SCANet network model, and the second predicted heat map corresponding to each human key point is obtained; a model loss function corresponding to the SCANet network model is generated according to the first loss function and the second loss function, and the SCANet network model is trained according to the model training samples with sample labels, and the trained SCANet network model is used as a posture recognition model.

[0119] Combination Figure 7 As shown, the key point groups corresponding to the key points of the human body include at least one of key point group A, key point group B, key point group C and key point group D, wherein key point group A = {left shoulder area, left elbow area, left wrist area}, key point group B = {right shoulder area, right elbow area, right wrist area}, key point group C = {left hip area, left knee area, left foot area}, and key point group D = {right hip area, right knee area, right foot area}.

[0120] In some embodiments, the model loss function corresponding to the SCANet network model is established based on the MSE loss model.

[0121] In some embodiments, a first loss model for key points of a human body is established based on the MSE loss model. The first loss model is as follows:

[0122]

[0123] In formula (4), n is the number of key points of the human body, P nis the first predicted heat map corresponding to the nth human key point, G n It is the first real heat map corresponding to the nth human key point.

[0124] In some embodiments, since the first loss function predicts the position of each key point separately, it does not take into account the correlation between the key points in the human body structure. For example, the left wrist area needs to be jointly calculated with the left elbow area and the left shoulder area for loss; through the key point group, adjacent human body key points are combined, and the second predicted heat map and the second real heat map corresponding to the human body key points are combined to calculate the model loss to obtain the second loss function.

[0125] In some embodiments, the model loss function corresponding to the SCANet network model is as follows:

[0126]

[0127] In formula (5), P Sn is the second predicted heat map corresponding to the nth human key point, G Sn It is the second real heat map corresponding to the nth human key point.

[0128] In some embodiments, since the head region and the neck region have a low correlation with other body regions and the head region and the neck region usually do not perform interactive tasks, the head region and the neck region do not participate in the model loss function.

[0129] In this way, compared with model training only through the model loss function corresponding to the key points of the human body, the association between the key points of the human body is combined to improve the posture recognition model's perception of the association between adjacent associated points, thereby improving the model's key point positioning ability, and further improving the model's ability to estimate human posture under occlusion.

[0130] Optionally, the posture recognition model performs key point recognition in the following manner: inputting the image area corresponding to the target object into the posture recognition model as a feature map; adjusting the number of channels and / or spatial resolution of the feature map through multiple SCAneck modules to extract features from the feature map; extracting features from the feature map through multiple SCAblock modules, wherein each SCAblock module corresponds to a different resolution and each SCAblock module corresponds to a different number of channels; adjusting the reduction layer position and the expansion layer position in the posture recognition model through the Sandglass module; and outputting the feature map with attention weights through the CoordAttention module.

[0131] Combination Figure 8As shown in the figure, the SCANet network model is improved based on the HRNet network model, where the HRNet network model consists of a Bottleneck module and a Basicblock module, while the SCANet network model is redesigned and a lighter SCAneck module and SCAblock module are proposed; at the same time, the Sandglass module replaces the standard 3×3 convolution and reconsiders the scaling coefficients of the reduction layer and the expansion layer, and adds the cross-channel feature information and precise spatial position information obtained through the Sandglass module to highlight the features of the smaller scale and occluded human key points in the image; finally, following the principle that the number of input channels is the same as the number of output channels, that is, inside the basic module of the SCANet network model, the number of channels of the Sandglass module and the CoordAttention module is set to C in =C out , which ensures minimal overhead for system access and maximum speed for the model.

[0132] In some embodiments, in order to prevent the loss of spatial position information of human key points in the feature map due to width downsampling operations and make it difficult to effectively perform multi-scale feature fusion of feature maps of different scales, the structure of the SCANet network model is divided into four stages: Stage 1, Stage 2, Stage 3 and Stage 4, and parallel branches with different numbers of channels and resolutions are used in each stage.

[0133] In some embodiments, the feature map is preprocessed by two 3×3 convolution kernels to standardize the resolution and number of channels of the feature map.

[0134] Combination Fig. 9 As shown in the figure, the preprocessed feature map is used as the input of the Stage 1 stage, and the number of channels and / or spatial resolution of the feature map are adjusted through four SCAneck modules to extract features from the feature map.

[0135] Combination Fig. 9 As shown, in the subsequent Stage 2, Stage 3 and Stage 4 stages, SCAblock modules with different resolutions and different numbers of channels are used to extract features from the feature map, where the resolutions are 1 / 4, 1 / 8, 1 / 16, and 1 / 32, and the numbers of channels are C, 2C, 3C, and 4C, respectively, with C=32.

[0136] In some embodiments, the parameter calculation formulas of the HRNet basic module and the SCANet basic module are as follows:

[0137]

[0138] In formula (6), P Bottleneck and P Basicblock are the parameters corresponding to the two basic modules in the HRNet network model, C in is the input channel of the current module, C mid is the intermediate feature channel of the current module, C out is the output channel of the current module, P SCAneck and P SCAblock are the parameters corresponding to the two basic modules in the SCANet network model, P CA is the parameter corresponding to the CoordAttention module, P SA It is the parameter quantity corresponding to the Sandglass module.

[0139] In some embodiments, compared with the basic module of the HRNet network model, the ratio of the SCAneck module and the SCAblock module of the SCANet network model to reduce the number of parameters during model training is:

[0140]

[0141] In formula (7), r neck1 is the ratio of the number of parameters reduced by the first SCAneck module compared to the Bottleneck module, r neck2 is the ratio of the number of parameters reduced by the following three SCAneck modules compared with the Bottleneck module, r block is the ratio of the number of parameters reduced by the SCAblock module compared to the Bottleneck module, and t is the preset scaling factor.

[0142] In some embodiments, based on equations (6) and (7), in Stage 1, since the number of channels of the first SCAneck module is different from that of the subsequent SCAneck modules, the input channel C of the first SCAneck module is in =64 and output channel C out =256, so when performing skip connection, a 1×1 convolution is added to convert the number of channels, so that the number of channels changes from 64 to 256, and the input channels and output channels of the other three SCAneck modules are all 256, so when performing skip connection, only feature maps are added without channel conversion. Therefore, the first SCAneck module reduces the number of parameters compared to the Bottleneck module, and r neck1 With r neck2 The results are different.

[0143] In some embodiments, based on equations (6) and (7), the SCAblock module is applied to Stage 2, Stage 3, and Stage 4. The SCAblock module has 32 32-channel, 32 64-channel, 28 128-channel, and 12 256-channel channels, and the input channel in the SCAblock module is equal to the output channel. Since the scaling factor set by the Sandglass module does not convert the number of channels inside the SCAblock module, the ratio of the parameter quantity reduced by the SCAblock modules of different parallel branches can be expressed as r. block Express.

[0144] Combination Fig.10 As shown in the figure, the inverted residual module is reversed by the Sandglass module, and two 1×1 convolutions are added in the middle of the deep convolution to adjust the positions of the reduction layer and the expansion layer. This fundamentally solves the problem that when mapping from high-dimensional features to low-dimensional features, all useful feature information cannot be retained due to channel compression; in order to save more feature channels and spatial context information, jump connections are created between high-dimensional features; if the number of input and output channels is not equal, the jump connection will not be used.

[0145] In some embodiments, the parameter quantity and operation complexity calculation formula corresponding to the Sandglass module are as follows:

[0146]

[0147] In formula (8), a is the preset scaling factor, G SA is the computational complexity corresponding to the Sandglass module, H is the height of the feature map, and W is the width of the feature map.

[0148] Combination Fig.11 As shown in the figure, the CoordAttention module not only obtains the feature information between channels, but also captures the precise location information and the area of ​​interest in the spatial direction. The CoordAttention module includes a Coordinate Information Embedding Module and a CoordinateAttention Generation Module. First, an adaptive average pooling layer is used to extract the features of each channel of the input feature map in the vertical and horizontal directions. Then, the feature maps are concatenated and a 1×1 convolution is used to generate a feature map with vertical and horizontal spatial information. Then, the feature map is divided into two feature maps along the spatial direction and a 1×1 convolution is used to convert the number of channels. Finally, the attention weights in the vertical and horizontal spatial directions are obtained, and they are multiplied with the input feature map to output the final feature map.

[0149] In some embodiments, the CoordAttention module uses a total of three 1×1 convolutions. The parameter quantity and computational complexity calculation formula corresponding to the CoordAttention module are as follows:

[0150]

[0151] In formula (9), G CA is the computational complexity corresponding to the CoordAttention module.

[0152] Combination Fig.12 As shown, an embodiment of the present disclosure provides an interactive task state determination system, including an acquisition module 1201 , a first determination module 1202 , a second determination module 1203 and a third determination module 1204 .

[0153] The acquisition module 1201 is used to acquire a job image corresponding to an interactive job node, wherein the interactive job node is obtained by splitting the interactive task according to the interaction type, and the job image includes image areas corresponding to multiple job objects respectively;

[0154] The first determination module 1202 is used for obtaining the area of ​​each image area respectively corresponding to each image area, and calculating the area intersection and union ratio between the areas of each area, so as to determine the node state corresponding to the interactive operation node according to the area intersection and union ratio, if the interaction type corresponding to the interactive operation node includes an interactive behavior;

[0155] The second determination module 1203 is used for, if the interaction type corresponding to the interactive operation node includes independent behavior, performing key point recognition on the image area corresponding to the target object to obtain the object key point, and performing feature extraction according to the key point coordinates corresponding to the object key point to obtain the object feature vector, so as to determine the node state corresponding to the interactive operation node according to the object feature vector, wherein the target object includes at least a part of the operation object;

[0156] The third determining module 1204 is used to determine the task state corresponding to the interactive task according to the node state corresponding to each interactive operation node in the interactive task.

[0157] The interactive task state determination system provided by the embodiment of the present disclosure is adopted, by splitting the interactive task into interactive operation nodes corresponding to interactive behaviors or independent behaviors, and obtaining the operation images corresponding to the interactive operation nodes, the node state corresponding to the interactive behavior is determined according to the area intersection and union ratio between the operation objects in the image regions, and the object feature vector is determined according to the key points of the operation object, so as to determine the node state corresponding to the independent behavior according to the object feature vector. In this way, compared with setting the visual algorithm for the interactive task as a whole interactive behavior, the interactive task is divided into interactive operation nodes corresponding to the interactive behavior or independent behavior according to the interaction type, and the node state of each interactive operation node is judged respectively, and then the node state is judged according to the interaction type to focus on the specific behavior or object state, so that the state judgment is more detailed and accurate, and then the completion of the interactive task is accurately judged according to all the node states of the interactive task, thereby improving the accuracy of task state monitoring.

[0158] An embodiment of the present disclosure further provides an electronic device, including: a processor and a memory; the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the electronic device executes the above method.

[0159] Fig.13 The structure diagram of the computer system suitable for implementing the electronic device of the embodiment of the present application is shown. It should be noted that: Fig.13 The computer system 1300 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0160] like Fig.13 As shown, the computer system 1300 includes a central processing unit (CPU) 1301, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 1302 or the program loaded from the storage part 1308 to the random access memory (RAM) 1303, such as executing the method in the above embodiment. In the random access memory 1303, various programs and data required for system operation are also stored. The central processing unit 1301, the read-only memory 1302 and the random access memory 1303 are connected to each other through a bus 1304. An input / output (I / O) interface 1305 is also connected to the bus 1304.

[0161] The following components are connected to the input / output interface 1305: an input section 1306 including a keyboard, a mouse, etc.; an output section 1307 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1308 including a hard disk, etc.; and a communication section 1309 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1309 performs communication processing via a network such as the Internet. A drive 1310 is also connected to the input / output interface 1305 as needed. A removable medium 1311, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1310 as needed so that a computer program read therefrom is installed into the storage section 1308 as needed.

[0162] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 1309, and / or installed from a removable medium 1311. When the computer program is executed by a central processing unit (CPU) 1301, various functions defined in the system of the present application are executed.

[0163] The above description and the accompanying drawings fully illustrate the embodiments of the present disclosure so that those skilled in the art can practice them. Other embodiments may include structural, logical, electrical, process and other changes. The embodiments represent possible changes only. Unless explicitly required, separate components and functions are optional, and the order of operation can be changed. Parts and sub-samples of some embodiments may be included in or replace parts and sub-samples of other embodiments. Moreover, the words used in this application are only used to describe the embodiments and are not used to limit the claims. As used in the description of the embodiments and the claims, unless the context clearly indicates, the singular forms of "a", "an" and "the" are intended to include plural forms as well. Similarly, the term "and / or" as used in this application refers to any and all possible combinations of listings containing one or more associated ones. In addition, when used in the present application, the term "comprise" and its variants "comprises" and / or comprising refer to the presence of a stated sub-sample, whole, step, operation, element, and / or component, but do not exclude the presence or addition of one or more other sub-samples, wholes, steps, operations, elements, components and / or groups of these. In the absence of further restrictions, the elements defined by the sentence "comprising a ..." do not exclude the presence of other identical elements in the process, method or device comprising the elements. In this article, each embodiment may focus on the differences from other embodiments, and the same and similar parts between the various embodiments may refer to each other. For the methods, products, etc. disclosed in the embodiments, if they correspond to the method part disclosed in the embodiments, then the relevant parts can refer to the description of the method part.

[0164] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software may depend on the specific application and design constraints of the technical solution. Technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of the present disclosure. Technicians can clearly understand that for the convenience and simplicity of description, the specific working process of the above-described system, device and unit can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0165] In the embodiments disclosed herein, the disclosed methods and products (including but not limited to devices, equipment, etc.) can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units can be only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some sub-samples can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms. The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to implement this embodiment. In addition, each functional unit in the embodiment of the present disclosure may be integrated in a processing unit, or each unit may exist physically alone, or two or more units may be integrated in one unit.

[0166] The flowchart and block diagram in the accompanying drawings show the possible architecture, functions and operations of the system, method and computer program product according to the embodiment of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and a part of the module, program segment or code contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. In the description corresponding to the flowchart and block diagram in the accompanying drawings, the operations or steps corresponding to different boxes can also occur in a different order from the order disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. Each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified functions or actions, or may be implemented by a combination of dedicated hardware and computer instructions.

Claims

1. A method for determining an interactive task state, characterized in that: include: Acquire a job image corresponding to an interactive job node, wherein the interactive job node is obtained by splitting the interactive task according to the interaction type, and the job image includes image areas corresponding to a plurality of job objects; If the interaction type corresponding to the interactive operation node includes interactive behavior, obtaining the area corresponding to each of the image regions, and calculating the area intersection-to-union ratio between the area of ​​each of the regions, so as to determine the node state corresponding to the interactive operation node according to the area intersection-to-union ratio; If the interaction type corresponding to the interactive operation node includes independent behavior, key point recognition is performed on the image area corresponding to the target object to obtain object key points, and feature extraction is performed based on the key point coordinates corresponding to the object key points to obtain an object feature vector, so as to determine the node state corresponding to the interactive operation node based on the object feature vector, wherein the target object includes at least a portion of the operation object; The task status corresponding to the interactive task is determined according to the node status corresponding to each interactive operation node in the interactive task.

2. The method according to claim 1, characterized in that The working object includes at least a portion of a human object, a workpiece object, and a workbench object, and the method further includes at least one of the following: If the node type includes interactive behavior, and the image area corresponding to the workpiece object is outside the image area corresponding to the workbench object, determining the node state corresponding to the interactive operation node as an unfinished state, wherein the workbench object is used to carry the workpiece object; If the node type includes interactive behavior, and if the object key point corresponding to the human object is located outside the image area corresponding to the workpiece object, determining the node state corresponding to the interactive operation node as an unfinished state; If the object category corresponding to the workpiece object does not belong to a preset category, the node state corresponding to the interactive operation node is determined to be an unfinished state, wherein the object category is obtained by performing object recognition on the workpiece object in the operation image.

3. The method according to claim 1, characterized in that Perform key point recognition on the image area corresponding to the target object to obtain the object key points, including: If the target object is a human body object, acquiring a plurality of first human body images; Labeling key points of a human body in the first human body image to obtain sample labels, wherein the key points of the human body include at least one of an elbow area, a wrist area, and a shoulder area; Determining an occluded area in which key points of the human body are present in the first human body image, and replacing at least a portion of the occluded area in the first human body image with a preset image to be replaced, to obtain a second human body image, wherein no key points of the human body are present in the image to be replaced; The first human body image and / or the second human body image are used as model training samples, and model training is performed based on the model training samples with sample labels to obtain a posture recognition model, wherein the posture recognition model is used to identify key points of the image area corresponding to the target object to obtain object key points.

4. The method according to claim 2, characterized in that Model training is performed based on model training samples with sample labels to obtain a posture recognition model, including: Obtaining a basic training model, wherein the basic training model includes a SCANet network model, and the SCANet network model is composed of a SCAneck module, an SCAblock module, a Sandglass module, and a CoordAttention module; Generating a first loss function according to the first true heat map and the first predicted heat map, wherein the human body key points are labeled according to each of the model training samples to obtain first true heat maps corresponding to each of the human body key points, and performing target detection on the model training samples based on the human body key points using the SCANet network model to obtain first predicted heat maps corresponding to each of the human body key points; Obtaining association relationships between the human body key points, and dividing the human body key points into a plurality of key point groups according to the association relationships, wherein two human body key points in the same key point group and having an association relationship are mutually associated key points; Generating a second loss function according to the second true heat map and the second predicted heat map, wherein, according to each of the model training samples, the human body key points and the associated key points corresponding to the human body key points are labeled to obtain second true heat maps corresponding to each of the human body key points, and target detection is performed on the model training samples based on the human body key points and the associated key points corresponding to the human body key points through the SCANet network model to obtain second predicted heat maps corresponding to each of the human body key points; A model loss function corresponding to the SCANet network model is generated according to the first loss function and the second loss function, and the SCANet network model is trained according to the model training samples with sample labels, and the trained SCANet network model is used as a posture recognition model.

5. The method according to claim 3, characterized in that The posture recognition model performs key point recognition in the following way: Inputting the image area corresponding to the target object into the gesture recognition model as a feature map; Adjusting the number of channels and / or spatial resolution of the feature map through multiple SCAneck modules to extract features from the feature map; Extracting features from the feature map using a plurality of SCAblock modules, wherein each of the SCAblock modules corresponds to a different resolution and a different number of channels; Adjusting the reduction layer position and the expansion layer position in the gesture recognition model through the Sandglass module; The CoordAttention module outputs a feature map with attention weights.

6. The method according to claim 3, characterized in that Feature extraction is performed based on the key point coordinates corresponding to the key points of the object to obtain an object feature vector, including: Calculating the distance between the key points of the objects having the associated relationship according to the key point coordinates corresponding to each of the key points of the objects to obtain an associated distance vector; Establishing a key point connection line according to the association relationship between the key points of each object, and calculating according to the key point coordinates corresponding to each key point of each object to obtain the association angle vector between the key point connection lines; An object feature vector is generated according to the associated distance vector and / or the associated angle vector.

7. The method according to any one of claims 1 to 6, characterized in that Obtaining a job image corresponding to the interactive job node includes: In response to executing the interactive operation node, performing image acquisition on each of the operation objects to obtain an operation image stream, wherein the operation image stream includes a plurality of image frames; Any image frame is determined as a working image, target detection is performed on each working object in the working image, and a ROI area obtained by the target detection is used as an image area corresponding to the working object.

8. The method according to any one of claims 1 to 6, characterized in that The method further comprises: Obtaining a current job flow, wherein the current job flow consists of a plurality of interactive tasks arranged in a preset execution order; Splitting each interactive task into a task start node and a task end node, and using the task start node and the task end node as an interactive operation node respectively; Determining the interactive operation node with the first priority as the target node according to the preset execution order; In response to a target node, obtaining a node state corresponding to the target node; If the node state corresponding to the target node is the completed state, then determining the interactive operation node that is one order after the target node as the new target node according to the preset execution order; The process execution state corresponding to the current operation process is generated according to the node state corresponding to each interactive operation node.

9. An interactive task status determination system, characterized in that: include: An acquisition module is configured to acquire a job image corresponding to an interactive job node, wherein the interactive job node is obtained by splitting the interactive task according to the interaction type, and the job image includes image areas corresponding to a plurality of job objects; a first determining module configured to, if the interaction type corresponding to the interactive operation node includes an interactive behavior, obtain the area corresponding to each of the image regions, and calculate an area intersection-and-union ratio between the area of ​​each of the regions, so as to determine a node state corresponding to the interactive operation node according to the area intersection-and-union ratio; a second determining module configured to, if the interaction type corresponding to the interactive operation node includes independent behavior, perform key point recognition on an image region corresponding to a target object to obtain object key points, perform feature extraction based on key point coordinates corresponding to the object key points to obtain an object feature vector, and determine a node state corresponding to the interactive operation node based on the object feature vector, wherein the target object includes at least a portion of an operation object; The third determining module is configured to determine a task status corresponding to the interactive task according to a node status corresponding to each interactive operation node in the interactive task.

10. An electronic device, characterized in that: include: processor and memory; The memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the electronic device performs the method according to any one of claims 1 to 8.

Images