Method for completing occlusion information and related device

Abstract

The application provides a method for completing occlusion information and related equipment. The method comprises the following steps: obtaining an unoccluded image and an occluded image at a current time; determining a first state model of a mechanical arm according to obtained mechanical arm joint data at the current time; determining a second state model of the mechanical arm according to obtained mechanical arm joint data at a reference time; determining a dynamic occlusion model according to the first state model and the second state model; determining a labeled image at the current time according to the dynamic occlusion model and the occluded image; and inputting the labeled image and the unoccluded image into a pre-trained occlusion completion model to determine a completed image at the current time. Through the dynamic occlusion model, the occluded image is processed, and the labeled image obtained after processing is input into the pre-trained occlusion completion model, and finally the completed image at the current time is obtained. Through the combination of the state models of the mechanical arm at multiple times, the completed image information is more accurate.

Description

Technical Field

[0001] This application relates to the fields of computer and communication technology, and in particular to a method and related equipment for completing obscured information. Background Technology

[0002] Related technologies typically achieve omnidirectional, unobstructed modeling by adding multiple cameras at different locations, but this approach undoubtedly increases equipment costs significantly. Furthermore, pose calibration and data synchronization between multiple cameras greatly increase the complexity of the task, causing serious inconvenience for equipment maintenance and users.

[0003] Therefore, the occluded parts are usually filled in to meet the needs. However, in general occluded images, moving objects will have certain edge blurring problems. The methods in related technologies only use the occluded image at the current moment to achieve the filling, which cannot accurately find the occlusion edge, and thus cannot accurately fill in the blurred edge. This results in low accuracy when filling in image information. Summary of the Invention

[0004] In view of this, the purpose of this application is to propose a method and related equipment for supplementing occlusion information.

[0005] To achieve the above objectives, this application provides a method for completing occlusion information, comprising: a method for completing occlusion information, characterized in that it includes:

[0006] Obtain the unoccluded image and the occluded image at the current moment;

[0007] The first state model of the robotic arm is determined based on the obtained joint data of the robotic arm at the current moment;

[0008] The second state model of the robotic arm is determined based on the joint data of the robotic arm at the obtained reference time.

[0009] Determine the dynamic occlusion model based on the first state model and the second state model;

[0010] The labeled image at the current moment is determined based on the dynamic occlusion model and the occlusion image;

[0011] The labeled image and the unoccluded image are input into a pre-trained occlusion completion model to determine the completed image at the current time.

[0012] In one possible implementation, determining the first state model of the robotic arm based on the acquired joint data of the robotic arm at the current moment includes:

[0013] Obtain data information for each joint of the robotic arm at the current moment; wherein, the data information at the current moment includes: first angle information;

[0014] The pose of the robotic arm at the current moment is determined based on the first angle information;

[0015] The first state model of the robotic arm is determined by matching the current pose of the robotic arm with its physical model.

[0016] In one possible implementation, the step of matching the robot arm's physical model based on its current pose to determine the robot arm's first state model further includes:

[0017] The first simplified model of the robotic arm is determined based on the current pose of the robotic arm; wherein the first simplified model includes at least one first robotic arm module;

[0018] Each first robotic arm module is matched with its corresponding robotic arm physical model to determine at least one first physical model to be combined.

[0019] The first state model of the robotic arm is determined based on the at least one first physical model to be combined.

[0020] In one possible implementation, the reference time includes: a historical time; the second state model includes a historical state model;

[0021] The step of determining the second state model of the robotic arm based on the obtained robotic arm joint data at the reference time further includes:

[0022] Acquire data information of each joint of the robotic arm at the historical moment; wherein, the data information at the historical moment includes: second angle information;

[0023] The pose of the robotic arm at a historical moment is determined based on the second angle information;

[0024] The robot arm's physical model is matched with its pose at the historical moment to determine the robot arm's historical state model.

[0025] In one possible implementation, the step of matching the robot arm's physical model with the robot arm's pose at the historical moment to determine the robot arm's historical state model further includes:

[0026] A second simplified model of the robotic arm is determined based on the pose of the robotic arm at the historical moment; wherein the second simplified model includes at least one second robotic arm module;

[0027] Each second robotic arm module is matched with its corresponding robotic arm physical model to determine at least one second physical model to be combined.

[0028] The historical state model of the robotic arm is determined based on at least one second physical model to be combined.

[0029] In one possible implementation, the reference time includes a future time; the second state model includes a future state model.

[0030] The step of determining the second state model of the robotic arm based on the obtained robotic arm joint data at the reference time further includes:

[0031] The acquired control commands of the robotic arm are parsed to determine the movement data;

[0032] The data information of each joint of the robotic arm at the future moment is determined based on the movement data; wherein, the data information at the future moment includes: third angle information;

[0033] The pose of the robotic arm at the future moment is determined based on the third angle information;

[0034] The future state model of the robotic arm is determined by matching its pose at the future moment with the physical model of the robotic arm.

[0035] In one possible implementation, the step of matching the robot arm's physical model with its pose at a future time to determine the robot arm's future state model further includes:

[0036] A simplified model of the robotic arm is determined based on the pose of the robotic arm at a future moment; wherein, the simplified model includes at least one robotic arm module;

[0037] Each third robotic arm module is matched with its corresponding robotic arm physical model to determine at least one third physical model to be combined.

[0038] The future state model of the robotic arm is determined based on at least one of the third physical models to be combined.

[0039] In one possible implementation, the reference time includes: a historical time and a future time; the second state model includes: a historical state model and a future state model.

[0040] The step of determining the second state model of the robotic arm based on the obtained robotic arm joint data at the reference time further includes:

[0041] Acquire data information of each joint of the robotic arm at the historical moment; wherein, the data information at the historical moment includes: second angle information;

[0042] The pose of the robotic arm at a historical moment is determined based on the second angle information;

[0043] The robot arm's physical model is matched with the robot arm's pose at the historical moment to determine the robot arm's historical state model.

[0044] The acquired control commands of the robotic arm are parsed to determine the movement data;

[0045] The data information of each joint of the robotic arm at the future moment is determined based on the movement data; wherein, the data information at the future moment includes: third angle information;

[0046] The pose of the robotic arm at the future moment is determined based on the third angle information;

[0047] The future state model of the robotic arm is determined by matching the pose of the robotic arm at the future time with the physical model of the robotic arm.

[0048] The second state model is determined based on the historical state model and the future state model.

[0049] In one possible implementation, determining the labeled image at the current moment based on the dynamic occlusion model and the occlusion image further includes:

[0050] The robot arm image at the current moment and the robot arm image at the reference moment are simulated based on the dynamic occlusion model; wherein, the reference moment includes: historical moment and / or future moment;

[0051] The robot arm image at the current moment, the robot arm image at the reference moment, and the occluded image at the current moment are superimposed to determine the image to be labeled;

[0052] Based on the image to be labeled, extract the current position of the robotic arm and the position of the robotic arm at the reference time;

[0053] The robot arm positions at the current time and at the reference time in the image to be labeled are assigned confidence levels to determine the labeled image.

[0054] In one possible implementation, the confidence labeling of the robotic arm position at the current moment and the robotic arm position at the reference moment in the image to be labeled further includes:

[0055] The confidence level of the current position of the robotic arm in the image to be labeled is marked as the first confidence level;

[0056] The confidence level of the robotic arm position at the reference time in the image to be labeled is marked as the second confidence level; wherein the first confidence level is less than the second confidence level.

[0057] Based on the same inventive concept, one or more embodiments of this specification also provide a device for supplementing occlusion information, including:

[0058] The acquisition module is configured to acquire the unoccluded image and the occluded image at the current moment;

[0059] The determination module is configured to determine the first state model of the robotic arm based on the acquired joint data of the robotic arm at the current moment.

[0060] The determination module is configured to determine the second state model of the robotic arm based on the robotic arm joint data obtained at the reference time.

[0061] The merging module is configured to determine a dynamic occlusion model based on the first state model and the second state model;

[0062] The annotation module is configured to determine the annotation image at the current moment based on the dynamic occlusion model and the occlusion image;

[0063] The occlusion completion module is configured to input the labeled image and the unoccluded image into a pre-trained occlusion completion model to determine the completed image at the current time.

[0064] Based on the same inventive concept, one or more embodiments of this specification also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method for completing occlusion information as described in any of the foregoing.

[0065] Based on the same inventive concept, one or more embodiments of this specification also provide a non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform any of the above-described methods for completing occlusion information.

[0066] As can be seen from the above, the method and related equipment for completing occlusion information provided in this application acquire an unoccluded image and an occluded image at the current moment. Based on the acquired robotic arm joint data at the current moment, a first state model of the robotic arm is determined. Based on the acquired robotic arm joint data at a reference moment, a second state model of the robotic arm is determined. The first and second state models are merged to obtain a dynamic occlusion model. Based on the dynamic occlusion model and the occluded image, a labeled image at the current moment is determined. The labeled image and the unoccluded image are input into a pre-trained occlusion completion model to determine the completed image at the current moment. For the occluded image to be completed, by acquiring the state models of the robotic arm at the current moment and the reference moment, and using the state models to label the occluded image with confidence, the influence of the blurred parts of the occlusion edges in the occluded image on image completion is effectively reduced. This improves the quality of the completed image obtained in the pre-trained occlusion completion model. Furthermore, only one viewpoint camera is needed in this process, effectively reducing the complexity of the task and facilitating later use and equipment maintenance. Moreover, by combining the state models of the robotic arm at multiple moments, the completed image information is more accurate. Attached Figure Description

[0067] To more clearly illustrate the technical solutions in this application or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0068] Figure 1 This is a schematic diagram illustrating an application scenario of the method for supplementing occlusion information according to an embodiment of this application.

[0069] Figure 2 This is a flowchart illustrating the completion of occlusion information in an embodiment of this application.

[0070] Figure 3 This is a flowchart illustrating the process of determining the first state model of the robotic arm based on the acquired joint data of the robotic arm at the current moment, as described in an embodiment of this application.

[0071] Figure 4 This is a flowchart illustrating how the physical model of the robotic arm is matched with its current pose to determine a first state model of the robotic arm, as described in an embodiment of this application.

[0072] Figure 5 This is a flowchart illustrating the process of determining the future state model of a robotic arm based on the obtained joint data of the robotic arm at a reference time, as described in an embodiment of this application.

[0073] Figure 6This is a flowchart illustrating the process of determining the labeled image at the current moment based on the dynamic occlusion model and the occlusion image, as described in an embodiment of this application.

[0074] Figure 7 This is a schematic diagram of the device structure for supplementing occlusion information according to an embodiment of this application.

[0075] Figure 8 This is a schematic diagram of the electronic device structure according to an embodiment of this application. Detailed Implementation

[0076] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with specific embodiments and the accompanying drawings.

[0077] It should be noted that, unless otherwise defined, the technical or scientific terms used in the embodiments of this application should have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms "first," "second," and similar terms used in the embodiments of this application do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are only used to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0078] As described in the background section, the problem of camera occlusion by robotic arms in related technologies is generally solved by setting up multiple view cameras to achieve omnidirectional unobstructed modeling. By setting up multiple view cameras, human images from multiple angles can be obtained to achieve unobstructed modeling. However, setting up multiple cameras will increase the equipment cost significantly. In addition, setting up multiple cameras requires pose calibration and data synchronization during data processing, which also greatly increases the complexity of data processing and causes serious inconvenience to the use process and subsequent equipment maintenance.

[0079] Meanwhile, existing technologies also use the method of completing the occluded image of the robotic arm acquired by the camera to obtain the unoccluded human image. However, the applicant found through research that due to the delay in the camera's image acquisition, there will be blurred edges around the robotic arm in the occluded image acquired by the camera. However, when completing the occluded image, the existing technology only completes the occluded image using the initial unoccluded image. Due to the camera delay and the existence of blurred edges, the final completion result will be affected, resulting in low accuracy when completing the image information.

[0080] In summary, the method and related equipment for completing occlusion information provided in this application acquire an unoccluded image and an occluded image at the current moment. Based on the acquired robotic arm joint data at the current moment, a first state model of the robotic arm is determined. Based on the acquired robotic arm joint data at a reference moment, a second state model of the robotic arm is determined. The first and second state models are merged to obtain a dynamic occlusion model. An annotated image at the current moment is determined based on the dynamic occlusion model and the occluded image. The annotated image and the unoccluded image are input into a pre-trained occlusion completion model to determine the completed image at the current moment. For the occluded image to be completed, by acquiring the state models of the robotic arm at the current and reference moments and using the state models to annotate the confidence level of the occluded image, the influence of blurred occlusion edges on image completion is effectively reduced, thereby improving the quality of the completed image obtained in the pre-trained occlusion completion model. Furthermore, only one viewpoint camera is needed in this process, effectively reducing the complexity of the task and facilitating later use and equipment maintenance. Moreover, by combining the state models of the robotic arm at multiple moments, the completed image information is more accurate.

[0081] refer to Figure 1 This is a schematic diagram illustrating an application scenario for completing occlusion information provided in this application embodiment. The application scenario includes a terminal device 101, a server 102, and a data storage system 103. The terminal device 101, server 102, and data storage system 103 can all be connected via wired or wireless communication networks. The terminal device 101 includes, but is not limited to, desktop computers, mobile phones, mobile computers, tablets, media players, smart wearable devices, personal digital assistants (PDAs), or other electronic devices capable of performing the aforementioned functions. The server 102 and data storage system 103 can both be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0082] Server 102 is used to provide occlusion information completion services to users of terminal device 101. Terminal device 101 has a client installed that communicates with server 102. Users can input an unoccluded image and an occluded image at the current moment through the client. After clicking the completion button, the client sends the unoccluded image and the occluded image at the current moment to server 102. Server 102 acquires the robotic arm joint data at the current moment and a reference moment and determines a dynamic occlusion model. Based on the dynamic occlusion model and the occluded image at the current moment, it determines the labeled image at the current moment. The labeled image and the unoccluded image are input into the trained occlusion completion model to obtain the completed image corresponding to the occluded image at the current moment output by the occlusion completion model. The completed image is then sent to the client, and the client displays the completed image to the user to help the user complete the occluded image.

[0083] The data storage system 103 stores a large amount of training data. Each training data set includes an occluded image at the current moment, a dynamic occlusion model corresponding to the occluded image, and an unoccluded image. The server 102 can train the occlusion completion model based on the large amount of training data, enabling the occlusion completion model to complete the input occluded image. The sources of the training data include, but are not limited to, existing databases, data crawled from the Internet, or data uploaded by users when using the client. When the accuracy of the occlusion completion model output reaches a certain requirement, the server 102 can provide occlusion completion services to users based on the occlusion completion model. At the same time, the server 102 can continuously optimize the occlusion completion model based on newly added training data.

[0084] The following is combined Figure 1 The application scenarios described above illustrate the training method for the occlusion completion model and the method for completing occlusion information according to exemplary embodiments of this application. It should be noted that the above application scenarios are shown only to facilitate understanding of the spirit and principles of this application, and the embodiments of this application are not limited in any way in this regard. Rather, the embodiments of this application can be applied to any applicable scenario.

[0085] The method for completing occlusion information provided in this application will be specifically described below through specific embodiments.

[0086] refer to Figure 2 The method for supplementing occlusion information according to embodiments of this application includes the following steps:

[0087] Step S201: Obtain the unoccluded image and the occluded image at the current moment.

[0088] Step S202: Determine the first state model of the robotic arm based on the obtained joint data of the robotic arm at the current moment.

[0089] Step S203: Determine the second state model of the robotic arm based on the obtained robotic arm joint data at the reference time.

[0090] Step S204: Determine the dynamic occlusion model based on the first state model and the second state model.

[0091] Step S205: Determine the labeled image at the current moment based on the dynamic occlusion model and the occlusion image.

[0092] Step S206 inputs the labeled image and the unoccluded image into a pre-trained occlusion completion model to determine the completed image at the current time.

[0093] In some embodiments, the unoccluded image can be a human body image captured by the viewpoint camera without the robotic arm obstructing the view, when the robotic arm is not in operation. The purpose of acquiring the unoccluded image is to provide ground truth for subsequent completion of the occluded image. The occluded image at the current moment can be a human body image with the robotic arm obstructing the view, captured by the viewpoint camera at any time while the robotic arm is in operation; this image is the image to be completed.

[0094] It should be noted that although the embodiments of this application use human images as an example, the restoration of occluded images in this application is aimed at objects occluded by robotic arms. Those skilled in the art can undoubtedly understand that the object can be a human body or other items. The principle of completion is the same. Therefore, those skilled in the art should be able to adapt this application to application scenarios where the occluded object is not a human body.

[0095] While the view camera acquires the occlusion image at the current moment, it also acquires the joint data of the robotic arm at the same current moment, and determines the first state model of the robotic arm based on the joint data.

[0096] refer to Figure 3 This application embodiment determines the first state model of the robotic arm based on the obtained current robotic arm joint data, including the following steps:

[0097] Step S301: Obtain data information of each joint of the robotic arm at the current moment; wherein, the data information at the current moment includes: first angle information.

[0098] Step S302: Determine the pose of the robotic arm at the current moment based on the first angle information.

[0099] Step S303: Match the physical model of the robotic arm with the pose of the robotic arm at the current moment to determine the first state model of the robotic arm.

[0100] In some embodiments, the robotic arm may have at least one joint. By acquiring the current arm length, coordinate information of each joint, and angle information of the joints—the above information can be acquired in real time by the robotic arm's control system—the coordinate values ​​of each joint of the robotic arm and the length and angle of the arm in space can be obtained, thereby determining the current pose of the robotic arm.

[0101] That is, for example, the robotic arm at the current moment can have a forearm and a large arm. If the angle between the forearm joint and the large arm joint at the current moment is 90°, and the coordinate information of the forearm shows that its extension direction is perpendicular to the ground, then the pose of the robotic arm at the current moment can be a bent large arm and a forearm perpendicular to the ground and pointing upwards, presenting a raised hand posture.

[0102] It should be noted that although the above embodiments use a raised hand pose as an example, those skilled in the art will undoubtedly understand that the robotic arm may have more than one forearm and one upper arm, and the pose may not be limited to the raised hand pose shown in the above embodiments. Furthermore, the length of the robotic arm at the current moment, the coordinate information of each joint, and the angle information of the arm can vary. That is, the final obtained pose of the robotic arm at the current moment can also have multiple results. Similarly, the following methods for obtaining the pose at historical moments, future moments, and both historical moments and future moments are similar to the above methods for obtaining the pose of the robotic arm at the current moment, and all can have multiple results.

[0103] After obtaining the pose of the robotic arm at the current moment, the physical model of the robotic arm corresponding to the pose is matched to obtain the first state model.

[0104] For details, please refer to Figure 4 The method of determining the first state model of the robotic arm by matching its physical model with its current pose in this embodiment includes the following steps:

[0105] Step S401: Determine the first simplified model of the robotic arm based on the current pose of the robotic arm; wherein the first simplified model includes at least one first robotic arm module.

[0106] Step S402: Match the physical model of the robotic arm corresponding to each of the first robotic arm modules to determine at least one first physical model to be combined.

[0107] Step S403: Determine the first state model of the robotic arm based on the at least one first physical model to be combined.

[0108] In some embodiments, to simplify the calculation process, the robotic arm can be simplified into a point-bar structure, with the joints of the robotic arm considered as points and the arms as bars. Based on the current pose of the robotic arm calculated in the above steps, the angles of the robotic arm joints, the length and angle of the arms in the pose are mapped onto a first simplified model, resulting in a first simplified model reflecting the pose of the robotic arm. However, since the first simplified model is not what obstructs the human body during actual use, it is necessary to match the first simplified model with the corresponding physical model of the robotic arm. Specifically, the joints corresponding to the angles of the points in the first simplified model and the arms corresponding to the angles and lengths of the bars in the first simplified model are searched in the robotic arm database. Then, the points in the first simplified model are replaced with the corresponding joints, and the bars are replaced with the corresponding arms. Finally, the aforementioned joints and arms are combined to obtain the first state model of the robotic arm.

[0109] In some embodiments, after calculating the pose of the robotic arm at the current moment, the robotic arm can be simplified into a point-bar structure, and a first simplified model can be determined accordingly. After determining the first simplified model, the first simplified model is treated as a whole, and a robotic arm in the robotic arm database that corresponds to the angle of the point in the simplified model and the length and angle of the bar are corresponding is searched. Then, the first simplified model is replaced by the robotic arm to obtain the first state model of the robotic arm.

[0110] In some embodiments, a simulated robotic arm can be pre-set, with the same size as the robotic arm that obscures the human body in reality. After obtaining the data information of each joint of the robotic arm at the current moment in step S301, the joint data at the current moment is input into the system controlling the simulated robotic arm. The simulated robotic arm can automatically simulate the pose of the robotic arm at the current moment. Since the simulated robotic arm and the real robotic arm have the same proportion, the simulation result obtained after simulation is directly used as the first state model.

[0111] In some embodiments, while acquiring the data information of each joint of the robotic arm at the current moment, the data information of each joint of the robotic arm at a reference moment can also be acquired. The reference moment can be a historical moment and / or a future moment.

[0112] Furthermore, when the reference time is a historical time, the second state model can include a historical state model. Data information for each joint of the robotic arm at the historical time can be obtained—this data can be directly obtained from the historical data of the robotic arm control system. Specifically, the historical data includes the arm's length, the coordinates of each joint, and the angles of the joints. The coordinate values ​​of each joint, as well as the arm's length and angle in space, can be obtained, thus determining the robotic arm's pose at the historical time. After obtaining the robotic arm's pose at the historical time, the pose is matched to the robotic arm's physical model to obtain the historical state model.

[0113] In some embodiments, to simplify the calculation process, the robotic arm can be simplified into a point-rod structure, with the joints of the robotic arm considered as points and the arms as rods. Based on the pose of the robotic arm at a historical moment calculated in the above steps, the angles of the robotic arm joints, the length and angle of the arms in the pose are mapped onto a simplified model, resulting in a second simplified model reflecting the pose of the robotic arm. However, since the second simplified model is not what obstructs the human body during actual use, it is necessary to match the second simplified model with the corresponding physical model of the robotic arm. Specifically, the joints corresponding to the angles of the points in the simplified model and the arms corresponding to the angles and lengths of the rods in the simplified model are searched in the robotic arm database. Then, the points in the simplified model are replaced with the corresponding joints, and the rods are replaced with the corresponding arms. Finally, the aforementioned joints and arms are combined to obtain the historical state model of the robotic arm.

[0114] In some embodiments, after calculating the pose of the robotic arm at historical moments, the robotic arm can be simplified into a point-bar structure, and a second simplified model can be determined accordingly. After determining the simplified model, the simplified model is treated as a whole, and a robotic arm database is searched for robotic arms whose angles correspond to the points in the simplified model and whose bar lengths and angles correspond to the bar lengths. Then, the second simplified model is replaced entirely with the robotic arm to obtain the historical state model of the robotic arm.

[0115] In some embodiments, a simulated robotic arm can be pre-set, with the same size as the robotic arm that obscures the human body in reality. After acquiring the data information of each joint of the robotic arm at a historical moment, the joint data at the historical moment is input into the system that controls the simulated robotic arm. The simulated robotic arm can automatically simulate the posture of the robotic arm at the historical moment. Since the simulated robotic arm and the real robotic arm have the same proportion, the simulation result obtained after simulation is directly used as the historical state model.

[0116] In some embodiments, when the reference time is a future time, the second state model may include a future state model. Specifically, the second state model of the robotic arm may be determined by acquiring robotic arm joint data at a future time.

[0117] For details, please refer to Figure 5 The method for determining the second state model of the robotic arm based on the obtained joint data of the robotic arm at a reference time, as described in this embodiment, includes the following steps:

[0118] Step S501: Parse the obtained control commands of the robotic arm to determine the movement data.

[0119] Step S502: Determine the data information of each joint of the robotic arm at the future moment based on the movement data; wherein, the data information at the future moment includes: third angle information.

[0120] Step S503: Determine the pose of the robotic arm at the future moment based on the third angle information.

[0121] Step S504: Match the physical model of the robotic arm with the pose of the robotic arm at the future time to determine the future state model of the robotic arm.

[0122] In some embodiments, data information for each joint of the robotic arm at a future moment can be obtained from the control commands of the robotic arm. The control commands for the robotic arm can be preset before it begins operation.

[0123] It should be noted that the control commands for the robotic arm can be to move a point on the robotic arm from one coordinate point to another, or to move a point on the robotic arm from one angle to another. Therefore, the future motion state of the robotic arm's coordinate points or angles can be obtained from the control commands, i.e., the movement data mentioned in step S501.

[0124] Furthermore, once the motion data is acquired, the robot arm's data information at a specific future moment can be accurately calculated based on the data and existing algorithms. Specifically, this includes the future arm length, the coordinates of each joint, and the angles of the joints. This allows for the acquisition of the coordinate values ​​of each joint and the arm's length and angle in space, thus determining the robot arm's pose at the future moment. After obtaining the robot arm's pose at the future moment, the pose is matched with the robot arm's physical model to obtain the future state model.

[0125] In some embodiments, to simplify the calculation process, the robotic arm can be simplified into a point-bar structure, with the joints of the robotic arm considered as points and the arms as bars. Based on the future pose of the robotic arm calculated in the above steps, the angles of the robotic arm joints, the length and angle of the arms in the pose are mapped onto a third simplified model, resulting in a third simplified model reflecting the pose of the robotic arm. However, since the third simplified model is not what obstructs the human body during actual use, it is necessary to match the third simplified model with the corresponding physical model of the robotic arm. Specifically, the joints corresponding to the angles of the points in the third simplified model and the arms corresponding to the angles and lengths of the bars in the simplified model are searched in the robotic arm database. Then, the points in the third simplified model are replaced with the corresponding joints, and the bars are replaced with the corresponding arms. Finally, the aforementioned joints and arms are combined to obtain the future state model of the robotic arm.

[0126] In some embodiments, after calculating the pose of the robotic arm at a future moment, the robotic arm can be simplified into a point-bar structure, and a third simplified model can be determined accordingly. After determining the third simplified model, the third simplified model is treated as a whole, and a robotic arm database is searched for that corresponds to the angle of the point in the simplified model and the length and angle of the bar. Then, the third simplified model is replaced by the robotic arm to obtain the future state model of the robotic arm.

[0127] In some embodiments, a simulated robotic arm can be pre-set, with the same size as the robotic arm that obscures the human body in reality. After obtaining the data information of each joint of the robotic arm at a future moment, the joint data at the future moment is input into the system that controls the simulated robotic arm. The simulated robotic arm can automatically simulate the pose of the robotic arm at the future moment. Since the simulated robotic arm has the same proportion as the real robotic arm, the simulation result obtained after simulation is directly used as the future state model.

[0128] In some embodiments, when the reference time includes a historical time and a future time, the second state model may include a historical state model and a future state model. Specifically, the data information of each joint of the robotic arm at the historical time can be obtained. The data information of the historical time can be directly obtained from the historical data of the robotic arm control system. The data information of the historical time specifically includes the length of the arm, the coordinate information of each joint, and the angle information of the joint. The coordinate values ​​of each joint of the robotic arm and the length and angle of the arm in space can be obtained, thereby obtaining the pose of the robotic arm at the historical time.

[0129] After obtaining the pose of the robotic arm at a historical moment, the corresponding physical model of the robotic arm is matched to obtain the historical state model.

[0130] In some embodiments, to simplify the calculation process, the robotic arm is simplified into a point-bar structure, with the joints of the robotic arm considered as points and the arms as bars. Based on the pose of the robotic arm at a historical moment calculated in the above steps, the angles of the robotic arm joints, the length and angle of the arms in the pose are mapped onto a simplified model to obtain a simplified model reflecting the pose of the robotic arm. However, since the simplified model is not what obstructs the human body during actual use, it is necessary to match the simplified model with the corresponding physical model of the robotic arm. Specifically, the joints corresponding to the angles of the points in the simplified model and the arms corresponding to the angles and lengths of the bars in the simplified model are searched in the robotic arm database. Then, the points in the simplified model are replaced with the corresponding joints, and the bars are replaced with the corresponding arms. Finally, the aforementioned joints and arms are combined to obtain the historical state model of the robotic arm.

[0131] In some embodiments, after calculating the pose of the robotic arm at historical moments, the robotic arm is similarly simplified into a point-bar structure, and a corresponding simplified model is determined. After determining the simplified model, the simplified model is treated as a whole, and a robotic arm database is searched for that corresponds to the angle of the point in the simplified model and the length and angle of the bar. Then, the simplified model is replaced by the robotic arm to obtain the historical state model of the robotic arm.

[0132] In some embodiments, a simulated robotic arm is pre-set, with the same size as the robotic arm that obscures the human body in reality. After acquiring the data information of each joint of the robotic arm at a historical moment, the joint data at the historical moment is input into the system controlling the simulated robotic arm. The simulated robotic arm can automatically simulate the posture of the robotic arm at the historical moment. Since the simulated robotic arm and the real robotic arm have the same proportion, the simulation result obtained after simulation is directly used as the historical state model.

[0133] In some embodiments, data information for each joint of the robotic arm at a future time is obtained from the control commands of the robotic arm. The control commands for the robotic arm are pre-set before it begins operation.

[0134] The control commands for a robotic arm are to move a point on the arm from one coordinate point to another, or to move a point on the arm from one angle to another. Therefore, the future motion state of the robotic arm's coordinate points or angles, i.e., movement data, can be obtained from the control commands.

[0135] Once the motion data is acquired, based on the relevant algorithms in the existing technology, the data information of the robotic arm at a certain future moment can be accurately calculated. Specifically, this includes the length of the arm at the future moment, the coordinate information of each joint, and the angle information of the joints. Then, the coordinate values ​​of each joint of the robotic arm, as well as the length and angle of the arm in space, can be obtained, thereby determining the pose of the robotic arm at the future moment.

[0136] After obtaining the pose of the robotic arm at a future moment, the physical model of the robotic arm corresponding to the pose is matched to obtain the future state model.

[0137] In some embodiments, to simplify the calculation process, the robotic arm is simplified into a point-bar structure, with the joints of the robotic arm considered as points and the arms as bars. Based on the future pose of the robotic arm calculated in the above steps, the angles of the robotic arm joints, the length and angle of the arms in the pose are mapped onto a simplified model to obtain a simplified model reflecting the pose of the robotic arm. However, since the simplified model is not what obstructs the human body during actual use, it is necessary to match the simplified model with the corresponding physical model of the robotic arm. Specifically, the joints corresponding to the angles of the points in the simplified model and the arms corresponding to the angles and lengths of the bars in the simplified model are searched in the robotic arm database. Then, the points in the simplified model are replaced with the corresponding joints, and the bars are replaced with the corresponding arms. Finally, the aforementioned joints and arms are combined to obtain the future state model of the robotic arm.

[0138] In some embodiments, after calculating the pose of the robotic arm at a future moment, the robotic arm is similarly simplified into a point-bar structure, and a simplified model is determined accordingly. After determining the simplified model, the simplified model is treated as a whole, and a robotic arm database is searched for that corresponds to the angle of the point in the simplified model and the length and angle of the bar. Then, the simplified model is replaced by the robotic arm to obtain the future state model of the robotic arm.

[0139] In some embodiments, a simulated robotic arm is pre-set, with the same size as the robotic arm that obscures the human body in reality. After obtaining the data information of each joint of the robotic arm at a future time, the joint data at the future time is input into the system that controls the simulated robotic arm. The simulated robotic arm can automatically simulate the pose of the robotic arm at the future time. Since the simulated robotic arm has the same proportion as the real robotic arm, the simulation result obtained after simulation is directly used as the future state model.

[0140] Furthermore, the historical state model and the future state model from the above steps can be combined into the second state model.

[0141] It should be noted that the number of historical moments and reference moments mentioned in the above steps is not limited to one; there can be multiple. Correspondingly, the number of state models in the second state model can also be multiple.

[0142] After determining the first-state model and the second-state model through the above steps, the data from the first-state model and the second-state model are summarized to determine the dynamic occlusion model. Furthermore, this dynamic occlusion model and the occluded image can be used to determine the labeled image at the current time.

[0143] For details, please refer to Figure 6 The method for determining the labeled image at the current moment based on the dynamic occlusion model and the occlusion image in this embodiment includes the following steps:

[0144] Step S601: Simulate the robot arm image at the current moment and the robot arm image at the reference moment based on the dynamic occlusion model; wherein, the reference moment includes: historical moment and / or future moment.

[0145] Step S602: Overlay the current robotic arm image, the reference robotic arm image, and the occlusion image at the current moment to determine the image to be labeled.

[0146] Step S603: Based on the image to be labeled, extract the current position of the robotic arm and the position of the robotic arm at the reference time.

[0147] Step S604: Confidence level annotation is performed on the current position of the robotic arm and the reference position of the robotic arm in the image to be annotated to determine the annotated image.

[0148] In some embodiments, a simulation system can be used to simulate the same viewpoint as a real-world camera. Under this viewpoint, a camera with the same viewpoint in the simulation environment is used to image the dynamic occlusion model in the simulation environment, thereby obtaining an image of the robotic arm at the current moment and an image of the robotic arm at a reference moment.

[0149] Furthermore, the current robotic arm image and the reference robotic arm image are superimposed on the current occlusion image to obtain the image to be labeled. The image to be labeled may include both the reference robotic arm image and the current robotic arm image. The overall image can be based on the current robotic arm image, and the position of the reference robotic arm image on the base image is depicted on the current robotic arm image.

[0150] It should be noted that the robotic arm image at the reference time may include robotic arm images at historical times and / or robotic arm images at future times. That is, the base image may only include the robotic arm position corresponding to the robotic arm in the robotic arm image at historical times or the robotic arm position corresponding to the robotic arm in the robotic arm image at future times, or it may include the robotic arm position corresponding to the robotic arm in both the robotic arm images at historical times and the robotic arm images at future times.

[0151] Furthermore, the robot arm positions at the current and reference times are extracted from the image to be labeled. Then, the confidence levels of the robot arm positions at the current and reference times are labeled. It should be noted that when there is an overlap between the robot arm positions at the reference and current times, the confidence level of the robot arm position at the current time is labeled first. Then, the overlapping part between the robot arm position at the reference time and the robot arm position at the current time is removed from the confidence level position at the reference time, and the confidence level of the remaining part is labeled.

[0152] In some embodiments, when assigning confidence levels, the confidence level of the robotic arm position at the current moment is lower than the confidence level of the robotic arm position at a reference moment. Furthermore, if multiple reference moments exist, the longer the time interval between a reference moment and the current moment, the higher the confidence level of the robotic arm position corresponding to that reference moment. That is, for example, if the current moment is 0 milliseconds, one reference moment A is -1 milliseconds, and another reference moment B is -5 milliseconds, then the confidence level of the robotic arm position corresponding to reference moment A is lower than the confidence level of the robotic arm position corresponding to reference moment B. Or, for example, if the current moment is 0 milliseconds, one reference moment C is -2 milliseconds, and another reference moment D is 10 milliseconds, then the confidence level of the robotic arm position corresponding to reference moment C is lower than the confidence level of the robotic arm position corresponding to reference moment D.

[0153] In some embodiments, the confidence level of the current position of the robotic arm can be set to zero.

[0154] After confidence-labeling the occluded image, the labeled image is obtained. Then, the labeled image and the unoccluded image are input into the pre-trained occlusion completion model to finally obtain the completed image at the current time.

[0155] In some embodiments, the pre-trained occlusion completion model uses the initial unoccluded image of the sample, the occluded image of the sample at the current time, the unoccluded image of the sample at the current time, and the dynamic occlusion model of the sample as the training set. During training, the dynamic occlusion model of the sample is used to label the occluded image with confidence to obtain the labeled image of the sample. Then, machine learning methods are used, with the labeled image of the sample and the initial unoccluded image of the sample as input, and the occlusion-completed image as output. The loss is calculated with the occlusion-completed image and the unoccluded image of the sample at the current time, and the occlusion completion model is obtained through iterative training.

[0156] In some embodiments, the training set of the pre-trained occlusion completion model is obtained in a simulation environment. The training set consists of real unoccluded images, occluded images at the current moment in the simulation environment, a dynamic occlusion model in the simulation environment, and unoccluded images at the current moment in the simulation environment. During training, the dynamic occlusion model in the simulation environment is used to label the occluded images at the current moment in the simulation environment with confidence, resulting in labeled images in the simulation environment. Then, machine learning methods are used, with the labeled images in the simulation environment and real unoccluded images as input, to obtain simulated occlusion completion images. The loss is calculated using simulated occlusion completion images and unoccluded images at the current moment in the simulation environment, and the occlusion completion model is obtained through iterative training.

[0157] As can be seen from the above embodiments, the method for completing occlusion information described in this application involves acquiring an unoccluded image and an occluded image at the current moment, determining a first state model of the robotic arm based on the acquired robotic arm joint data at the current moment, determining a second state model of the robotic arm based on the acquired robotic arm joint data at a reference moment, determining a dynamic occlusion model based on the first state model and the second state model, determining a labeled image at the current moment based on the dynamic occlusion model and the occluded image, and inputting the labeled image and the unoccluded image into a pre-trained occlusion completion model to determine the completed image at the current moment. For the occluded image to be completed, by acquiring the state model of the robotic arm at the current time and the reference time, and using the state model to label the confidence of the occluded image, the edge blurring phenomenon caused by the delay of the view camera can be ignored. Furthermore, the influence of the blurred part of the occluded edge in the occluded image on the image completion is effectively reduced, thereby improving the quality of the completed image obtained in the pre-trained occlusion completion model. In addition, only one view camera is needed in this process, which effectively reduces the complexity of the task and facilitates later use and equipment maintenance. Moreover, by combining the state model of the robotic arm at multiple times, the completed image information is more accurate.

[0158] It should be noted that the method in this embodiment can be executed by a single device, such as a computer or server. The method can also be applied in a distributed scenario, where multiple devices cooperate to complete the task. In such a distributed scenario, one of these devices may execute only one or more steps of the method in this embodiment, and the multiple devices will interact with each other to complete the method described.

[0159] It should be noted that the above description describes some embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the above embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0160] Based on the same inventive concept, corresponding to any of the above embodiments, this application also provides a device for supplementing occlusion information.

[0161] refer to Figure 7 The device for completing the occlusion information includes:

[0162] The acquisition module is configured to acquire the unoccluded image and the occluded image at the current moment;

[0163] The determination module is configured to determine the first state model of the robotic arm based on the acquired joint data of the robotic arm at the current moment.

[0164] The determination module is configured to determine the second state model of the robotic arm based on the robotic arm joint data obtained at the reference time.

[0165] The merging module is configured to determine a dynamic occlusion model based on the first state model and the second state model;

[0166] The annotation module is configured to determine the annotation image at the current moment based on the dynamic occlusion model and the occlusion image;

[0167] The occlusion completion module is configured to input the labeled image and the unoccluded image into a pre-trained occlusion completion model to determine the completed image at the current time.

[0168] In one possible implementation, the determining module is further configured as follows:

[0169] Obtain data information for each joint of the robotic arm at the current moment; wherein, the data information at the current moment includes: first angle information;

[0170] The pose of the robotic arm at the current moment is determined based on the first angle information;

[0171] The first state model of the robotic arm is determined by matching the current pose of the robotic arm with its physical model.

[0172] In one possible implementation, the device further includes: a matching module;

[0173] The matching module is further configured as follows:

[0174] The first simplified model of the robotic arm is determined based on the current pose of the robotic arm; wherein the first simplified model includes at least one first robotic arm module;

[0175] Each of the first robotic arm modules is matched with its corresponding physical model to determine at least one first physical model to be combined.

[0176] The first state model of the robotic arm is determined based on the at least one first physical model to be combined.

[0177] In one possible implementation, the reference time includes: a historical time; the second state model includes a historical state model;

[0178] The determining module is further configured as follows:

[0179] Acquire data information of each joint of the robotic arm at the historical moment; wherein, the data information at the historical moment includes: second angle information;

[0180] The pose of the robotic arm at a historical moment is determined based on the second angle information;

[0181] The robot arm's physical model is matched with its pose at the historical moment to determine the robot arm's historical state model.

[0182] In one possible implementation, the determining module is further configured as follows:

[0183] A second simplified model of the robotic arm is determined based on the pose of the robotic arm at the historical moment; wherein the second simplified model includes at least one second robotic arm module;

[0184] Each second robotic arm module is matched with its corresponding robotic arm physical model to determine at least one second physical model to be combined.

[0185] The historical state model of the robotic arm is determined based on at least one of the second physical models to be combined.

[0186] In one possible implementation, the reference time includes a future time; the second state model includes a future state model.

[0187] The determining module is further configured as follows:

[0188] The acquired control commands of the robotic arm are parsed to determine the movement data;

[0189] The data information of each joint of the robotic arm at the future moment is determined based on the movement data; wherein, the data information at the future moment includes: third angle information;

[0190] The pose of the robotic arm at the future moment is determined based on the third angle information;

[0191] The future state model of the robotic arm is determined by matching its pose at the future moment with the physical model of the robotic arm.

[0192] In one possible implementation, the determining module is further configured as follows:

[0193] A third simplified model of the robotic arm is determined based on the pose of the robotic arm at a future moment; wherein, the third simplified model includes at least one third robotic arm module;

[0194] Each third robotic arm module is matched with its corresponding robotic arm physical model to determine at least one third physical model to be combined.

[0195] The future state model of the robotic arm is determined based on at least one third physical model to be combined.

[0196] In one possible implementation, the reference time includes: a historical time and a future time; the second state model includes: a historical state model and a future state model.

[0197] The determining module is further configured as follows:

[0198] Acquire data information of each joint of the robotic arm at the historical moment; wherein, the data information at the historical moment includes: second angle information;

[0199] The pose of the robotic arm at a historical moment is determined based on the second angle information;

[0200] The robot arm's physical model is matched with the robot arm's pose at the historical moment to determine the robot arm's historical state model.

[0201] The acquired control commands of the robotic arm are parsed to determine the movement data;

[0202] The data information of each joint of the robotic arm at the future moment is determined based on the movement data; wherein, the data information at the future moment includes: third angle information;

[0203] The pose of the robotic arm at the future moment is determined based on the third angle information;

[0204] The future state model of the robotic arm is determined by matching the pose of the robotic arm at the future time with the physical model of the robotic arm.

[0205] The second state model is determined based on the historical state model and the future state model.

[0206] In one possible implementation, the annotation module is further configured as follows:

[0207] The robot arm image at the current moment and the robot arm image at the reference moment are simulated based on the dynamic occlusion model; wherein, the reference moment includes: historical moment and / or future moment;

[0208] The robot arm image at the current moment, the robot arm image at the reference moment, and the occluded image at the current moment are superimposed to determine the image to be labeled;

[0209] Based on the image to be labeled, extract the current position of the robotic arm and the position of the robotic arm at the reference time;

[0210] The robot arm positions at the current time and at the reference time in the image to be labeled are assigned confidence levels to determine the labeled image.

[0211] In one possible implementation, the annotation module is further configured as follows:

[0212] The confidence level of the current position of the robotic arm in the image to be labeled is marked as the first confidence level;

[0213] The confidence level of the robotic arm position at the reference time in the image to be labeled is marked as the second confidence level; wherein the first confidence level is less than the second confidence level.

[0214] For ease of description, the above devices are described in terms of function, divided into various modules. Of course, in implementing this application, the functions of each module can be implemented in one or more software and / or hardware.

[0215] The apparatus of the above embodiments is used to implement the corresponding method for completing occlusion information in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0216] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method for completing occlusion information as described in any of the above embodiments.

[0217] Figure 8 This embodiment illustrates a more specific hardware structure of an electronic device, which may include a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, memory 1020, input / output interface 1030, and communication interface 1040 are interconnected internally via the bus 1050.

[0218] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0219] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0220] The input / output interface 1030 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components within the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touchscreens, microphones, various sensors, etc., while output devices may include displays, speakers, vibrators, indicator lights, etc.

[0221] The communication interface 1040 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0222] Bus 1050 includes a pathway for transmitting information between various components of the device, such as processor 1010, memory 1020, input / output interface 1030, and communication interface 1040.

[0223] It should be noted that although the above-described device only shows the processor 1010, memory 1020, input / output interface 1030, communication interface 1040, and bus 1050, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this specification, and not necessarily all the components shown in the figures.

[0224] The electronic devices described above are used to implement the corresponding methods for completing occlusion information in any of the foregoing embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0225] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this application also provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute the method for completing occlusion information as described in any of the above embodiments.

[0226] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0227] The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to execute the method for completing occlusion information as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0228] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of this application (including the claims) is limited to these examples; within the framework of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the embodiments of this application as described above, which are not provided in the details for the sake of brevity.

[0229] Additionally, to simplify the description and discussion, and to avoid obscuring the embodiments of this application, the well-known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided drawings. Furthermore, the apparatus may be shown in block diagram form to avoid obscuring the embodiments of this application, and this also takes into account the fact that the details of the implementation of these block diagram apparatuses are highly dependent on the platform on which the embodiments of this application will be implemented (i.e., these details should be fully understood by those skilled in the art). While specific details (e.g., circuits) have been set forth to describe exemplary embodiments of this application, it will be apparent to those skilled in the art that the embodiments of this application can be implemented without these specific details or with variations thereof. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0230] Although this application has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed.

[0231] The embodiments of this application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of this application should be included within the protection scope of this application.

Claims

1. A method for completing occlusion information, characterized in that, include: Obtain the unoccluded image and the occluded image at the current moment; The unobstructed image is an image that is not obstructed by the robotic arm when the robotic arm is not in operation; The occlusion image at the current moment is an image that is occluded by the robotic arm while the robotic arm is in operation; The first state model of the robotic arm is determined based on the obtained joint data of the robotic arm at the current moment; The second state model of the robotic arm is determined based on the joint data of the robotic arm at the obtained reference time. Determine the dynamic occlusion model based on the first state model and the second state model; The labeled image at the current moment is determined based on the dynamic occlusion model and the occlusion image; the labeled image at the current moment is determined by confidence labeling of the robot arm position at the current moment and the robot arm position at the reference moment in the image to be labeled; the robot arm position at the current moment and the robot arm position at the reference moment are extracted from the image to be labeled; the image to be labeled is determined by superimposing the robot arm image at the current moment, the robot arm image at the reference moment, and the occlusion image at the current moment; the robot arm image at the current moment and the robot arm image at the reference moment are simulated based on the dynamic occlusion model; The labeled image and the unoccluded image are input into a pre-trained occlusion completion model to determine the completed image at the current time.

2. The method according to claim 1, characterized in that, The step of determining the first state model of the robotic arm based on the acquired joint data of the robotic arm at the current moment includes: Obtain data information for each joint of the robotic arm at the current moment; wherein, the data information at the current moment includes: first angle information; The pose of the robotic arm at the current moment is determined based on the first angle information; The first state model of the robotic arm is determined by matching the current pose of the robotic arm with its physical model.

3. The method according to claim 2, characterized in that, The step of matching the robot arm's physical model with its current pose to determine the robot arm's first state model further includes: The first simplified model of the robotic arm is determined based on the current pose of the robotic arm; wherein the first simplified model includes at least one first robotic arm module; Each first robotic arm module is matched with its corresponding robotic arm physical model to determine at least one first physical model to be combined. The first state model of the robotic arm is determined based on the at least one first physical model to be combined.

4. The method according to claim 1, characterized in that, The reference time includes: historical time; the second state model includes historical state model; The step of determining the second state model of the robotic arm based on the obtained robotic arm joint data at the reference time further includes: Acquire data information of each joint of the robotic arm at the historical moment; wherein, the data information at the historical moment includes: second angle information; The pose of the robotic arm at a historical moment is determined based on the second angle information; The robot arm's physical model is matched with its pose at the historical moment to determine the robot arm's historical state model.

5. The method according to claim 4, characterized in that, The step of matching the robot arm's physical model with its pose at the historical moment to determine the robot arm's historical state model further includes: A second simplified model of the robotic arm is determined based on the pose of the robotic arm at the historical moment; wherein the second simplified model includes at least one second robotic arm module; Each second robotic arm module is matched with its corresponding robotic arm physical model to determine at least one second physical model to be combined. The historical state model of the robotic arm is determined based on at least one second physical model to be combined.

6. The method according to claim 1, characterized in that, The reference time includes: a future time; the second state model includes: a future state model; The step of determining the second state model of the robotic arm based on the obtained robotic arm joint data at the reference time further includes: The acquired control commands of the robotic arm are parsed to determine the movement data; The data information of each joint of the robotic arm at the future moment is determined based on the movement data; wherein, the data information at the future moment includes: third angle information; The pose of the robotic arm at the future moment is determined based on the third angle information; The future state model of the robotic arm is determined by matching its pose at the future moment with the physical model of the robotic arm.

7. The method according to claim 6, characterized in that, The step of matching the robot arm's physical model with its pose at a future time to determine the robot arm's future state model further includes: A third simplified model of the robotic arm is determined based on the pose of the robotic arm at a future moment; wherein, the third simplified model includes at least one third robotic arm module; Each third robotic arm module is matched with its corresponding robotic arm physical model to determine at least one third physical model to be combined. The future state model of the robotic arm is determined based on at least one third physical model to be combined.

8. The method according to claim 1, characterized in that, The reference time includes: historical time and future time; the second state model includes: historical state model and future state model; The step of determining the second state model of the robotic arm based on the obtained robotic arm joint data at the reference time further includes: Acquire data information of each joint of the robotic arm at the historical moment; wherein, the data information at the historical moment includes: second angle information; The pose of the robotic arm at a historical moment is determined based on the second angle information; The robot arm's physical model is matched with the robot arm's pose at the historical moment to determine the robot arm's historical state model. The acquired control commands of the robotic arm are parsed to determine the movement data; The data information of each joint of the robotic arm at the future moment is determined based on the movement data; wherein, the data information at the future moment includes: third angle information; The pose of the robotic arm at the future moment is determined based on the third angle information; The future state model of the robotic arm is determined by matching the pose of the robotic arm at the future time with the physical model of the robotic arm. The second state model is determined based on the historical state model and the future state model.

9. The method according to claim 1, characterized in that, The reference time includes: historical time and / or future time.

10. The method according to claim 9, characterized in that, The step of assigning confidence scores to the current position of the robotic arm and the reference position of the robotic arm in the image to be labeled further includes: The confidence level of the current position of the robotic arm in the image to be labeled is marked as the first confidence level; The confidence level of the robotic arm position at the reference time in the image to be labeled is marked as the second confidence level; wherein the first confidence level is less than the second confidence level.

11. A device for supplementing occlusion information, characterized in that, include: The acquisition module is configured to acquire the unoccluded image and the occluded image at the current moment; The unobstructed image is an image that is not obstructed by the robotic arm when the robotic arm is not in operation; The occlusion image at the current moment is an image that is occluded by the robotic arm while the robotic arm is in operation; The determination module is configured to determine the first state model of the robotic arm based on the acquired joint data of the robotic arm at the current moment. The determination module is configured to determine the second state model of the robotic arm based on the robotic arm joint data obtained at the reference time. The merging module is configured to determine a dynamic occlusion model based on the first state model and the second state model; The annotation module is configured to determine the annotation image at the current moment based on the dynamic occlusion model and the occlusion image; the annotation image at the current moment is determined by confidence annotation of the current robotic arm position and the robotic arm position at the reference moment in the image to be annotated; the current robotic arm position and the robotic arm position at the reference moment are extracted from the image to be annotated; the image to be annotated is determined by superimposing the robotic arm image at the current moment, the robotic arm image at the reference moment, and the occlusion image at the current moment; the current robotic arm image and the robotic arm image at the reference moment are simulated based on the dynamic occlusion model. The occlusion completion module is configured to input the labeled image and the unoccluded image into a pre-trained occlusion completion model to determine the completed image at the current time.

12. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 10.

13. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1 to 10.

Images