Method, device and computer-readable medium for moving opening and closing components based on a mechanical arm

Abstract

The embodiments of the present disclosure disclose a method, device and computer-readable medium for moving an opening and closing component based on a robotic arm. A specific implementation of the method includes: obtaining initial motion trajectory information; training a preset motion trajectory prediction network; determining the collection time of the first position information as the first collection time; executing the following monitoring movement steps: inputting the first position information and the second position information into the motion trajectory prediction model; collecting movement verification position information after the preset time of the first collection time; determining the collection time for collecting the movement verification position information as the collection time to be updated; updating the collection time to be updated to the first collection time; moving the opening and closing component to the position corresponding to the mobile position information; updating the mobile position information to the second position information; and executing the monitoring movement step again. This implementation reduces the number of failed opening and closing movements of the opening and closing component and extends the service life of the robotic arm.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technology, and in particular to a method, device, and computer-readable medium for moving an opening and closing component based on a robotic arm. Background Art

[0002] With the development of digitalization and intelligence in mechanical manufacturing, robots and robotic arms have been widely used in life. The movement of opening and closing components based on robotic arms is a technology that uses robotic arms to open and close the opening and closing components (e.g., microwave oven doors with hinged structures, drawers with slide rail structures) in devices (e.g., microwave ovens, pull-out side cabinets). At present, when using robotic arms to open and close the opening and closing components, the method usually adopted is: based on the movement path planned when the device is in a fixed position, the opening and closing components in the fixed position device are opened and closed by using the robotic arms.

[0003] However, when the above method is used to open and close the opening and closing components in the device, the following technical problems often occur:

[0004] First, based on the movement path planned when the device is in a fixed position, the opening and closing components in the fixed position device are opened and closed by the robotic arm. When the relative position of the fixed position device and the robotic arm changes (for example, the robotic arm or the fixed position device is moved), the robotic arm may not be able to open and close the opening and closing components according to the movement path planned when the device is in a fixed position, increasing the number of failures of the robotic arm to open and close the opening and closing components. At the same time, when the relative position of the fixed position device and the robotic arm changes, opening and closing the opening and closing components based on the movement path planned when the device is in a fixed position may cause the device or the robotic arm to be damaged due to excessive force after relative movement between the robotic arm and the device, thereby shortening the service life of the robotic arm.

[0005] Continuing, in the process of adopting technical solutions to solve the above-mentioned technical problem 1, the following technical problems often occur: when training the preset motion trajectory prediction network, it is necessary to generate the motion trajectory information of the opening and closing components in the device during the opening or closing movement (the three-dimensional position information of a point in the opening and closing components at each moment) as training data. The solution to generate motion trajectory information is generally to collect the motion trajectory information of the opening and closing components during the opening or closing movement through laser scanning technology. However, the above-mentioned conventional solution generates the motion trajectory information of the opening and closing components during the opening or closing movement through laser scanning technology, and there are still the following problems:

[0006] Second, through laser scanning technology, the motion trajectory information of the opening and closing components during the opening or closing process is collected. Since laser scanning requires time to scan point by point, the fast-moving opening and closing components may have moved during the scanning process, which may cause data loss or a large degree of distortion, resulting in low accuracy of the collected motion trajectory information.

[0007] The above information disclosed in this Background section is only for enhancement of understanding of the background of the present disclosure concept and therefore it may contain information that does not form the prior art that is already known in this country to a person of ordinary skill in the art. Summary of the invention

[0008] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0009] Some embodiments of the present disclosure propose a method for moving an opening and closing component based on a robotic arm, an electronic device, and a computer-readable medium to solve one or more of the technical problems mentioned in the above background technology section.

[0010] In a first aspect, some embodiments of the present disclosure provide a method for moving an opening and closing component based on a robotic arm, the method comprising: obtaining initial motion trajectory information of an opening and closing component in a target device; based on the initial motion trajectory information, training a preset motion trajectory prediction network to obtain a trained motion trajectory prediction model; in response to receiving preset request information representing the opening and closing movement of the opening and closing component in the target device, collecting three-dimensional spatial position information of a first target point and three-dimensional spatial position information of a second target point in the target device, wherein the first target point is a point in a non-opening and closing component, and the second target point is a point in an opening and closing component; determining the collected three-dimensional spatial position information of the first target point and the collected three-dimensional spatial position information of the second target point as first position information and second position information, respectively; determining a collection time of the first position information as a first collection time; and based on the first position information, the second position information and the first collection time , perform the following monitoring movement steps: input the first position information and the second position information into the motion trajectory prediction model to obtain the mobile position information; after the preset time of the first acquisition time, control the preset acquisition device to acquire the position information of the first target point in the target device as the mobile verification position information; determine the acquisition time for acquiring the mobile verification position information as the acquisition time to be updated; update the acquisition time to be updated to the first acquisition time to update the first acquisition time; in response to determining that the mobile verification position information is the same as the first position information, control the preset mechanical arm to clamp the opening and closing component in the target device at the position corresponding to the second position information and control the above-mentioned preset mechanical arm to move the opening and closing component to the position corresponding to the mobile position information; update the mobile position information to the second position information to update the second position information; based on the first position information, the updated second position information and the updated first acquisition time, perform the above-mentioned monitoring movement steps again.

[0011] In the second aspect, some embodiments of the present disclosure provide an electronic device applied to a method for moving an opening and closing component based on a robotic arm, comprising: a preset robotic arm; one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by one or more processors, the one or more processors implement the method described in any implementation method of the above-mentioned first aspect.

[0012] In a third aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method described in any implementation manner of the first aspect is implemented.

[0013] The above-mentioned embodiments of the present disclosure have the following beneficial effects: through the opening and closing component moving method based on the mechanical arm of some embodiments of the present disclosure, the number of failures in opening and closing the opening and closing component is reduced, and the service life of the mechanical arm is extended. Specifically, the reasons for the increase in the number of failures in opening and closing the opening and closing component and the short service life of the mechanical arm are: based on the moving path planned when the device is in a fixed position, the mechanical arm is used to open and close the opening and closing component in the device at the fixed position. When the relative position of the device at the fixed position and the mechanical arm changes (for example, the mechanical arm or the device at the fixed position is moved), the mechanical arm may not be able to open and close the opening and closing component according to the moving path planned when it is in the fixed position, thereby increasing the number of failures of the mechanical arm to open and close the opening and closing component. At the same time, when the relative position of the device at the fixed position and the mechanical arm changes, based on the moving path planned when the device is in the fixed position, the opening and closing component is opened and closed, which may cause the mechanical arm and the device to be damaged due to excessive force after relative movement, thereby resulting in a short service life of the mechanical arm. Based on this, some embodiments of the present disclosure disclose a method for moving an opening and closing component based on a mechanical arm. First, the initial motion trajectory information of the opening and closing component in the target device is obtained. Thus, the initial motion trajectory information for training the preset motion trajectory prediction network can be obtained. Then, based on the above initial motion trajectory information, the preset motion trajectory prediction network is trained to obtain a trained motion trajectory prediction model. Thus, the mobile position information corresponding to the position to be moved next by the opening and closing component can be obtained in real time point by point according to the three-dimensional spatial position information of the second target point in the opening and closing component. Afterwards, in response to receiving the preset request information representing the opening and closing movement of the opening and closing component in the above target device, the three-dimensional spatial position information of the first target point and the three-dimensional spatial position information of the second target point in the above target device are collected, wherein the above first target point is a point in the non-opening and closing component, and the above second target point is a point in the opening and closing component. Then, the collected three-dimensional spatial position information of the first target point and the collected three-dimensional spatial position information of the second target point are respectively determined as the first position information and the second position information. Thus, the first position information and the second position information used to generate the mobile position information can be obtained. Afterwards, the collection time of the first position information is determined as the first collection time. Thus, the first acquisition time for monitoring the moving step can be obtained. Then, based on the first position information, the second position information and the first acquisition time, the following monitoring moving step is performed: the first position information and the second position information are input into the motion trajectory prediction model to obtain the moving position information. Thus, the moving position information corresponding to the position to be moved can be predicted in real time to obtain the moving position information representing the position to be moved next by the opening and closing component. Afterwards, after the preset time of the first acquisition time, the preset acquisition device is controlled to collect the position information of the first target point in the target device as the moving verification position information.Then, the collection time of collecting the mobile verification position information is determined as the collection time to be updated. Then, the collection time to be updated is updated to the first collection time to update the first collection time. Thus, the first collection time can be updated. Then, in response to determining that the mobile verification position information is the same as the first position information, the preset mechanical arm is controlled to clamp the opening and closing component in the target device at the position corresponding to the second position information, and the preset mechanical arm is controlled to move the opening and closing component to the position corresponding to the mobile position information. Thus, the preset mechanical arm can be used to move the opening and closing component to the position predicted in real time to be moved next. Then, the mobile position information is updated to the second position information to update the second position information. Thus, the second position information can be updated. Then, based on the first position information, the updated second position information and the updated first collection time, the above-mentioned monitoring movement step is performed again. Also because the monitoring movement is used in the process of opening and closing the opening and closing component in the device, the mobile position information corresponding to the position to which the opening and closing component is to move next is predicted point by point in real time according to the first position information and the second position information collected in real time. The preset robotic arm is used to move the opening and closing component to the position where the opening and closing component will move next in real time, thereby reducing the number of opening and closing movement failures caused by opening and closing the opening and closing component according to the planned moving path when it is in a fixed position, and reducing the possibility of damage to the equipment or the robotic arm due to excessive force after relative movement between the robotic arm and the equipment during the opening and closing movement according to the planned moving path, thereby extending the service life of the robotic arm. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0015] Figure 1 is a flow chart of some embodiments of the method for moving an opening and closing component based on a mechanical arm according to the present disclosure;

[0016] Figure 2 It is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0017] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0018] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0019] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0020] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0021] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0022] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0023] Figure 1 The process 100 of some embodiments of the method for moving an opening and closing component based on a mechanical arm according to the present disclosure is shown. The method for moving an opening and closing component based on a mechanical arm comprises the following steps:

[0024] Step 101: Acquire initial motion trajectory information of an opening and closing component in a target device.

[0025] In some embodiments, the execution subject (e.g., a computing device) of the method for moving an opening and closing component based on a mechanical arm can obtain the initial motion trajectory information of the opening and closing component in the target device through a wired connection or a wireless connection. Among them, the above-mentioned target device can be a device with an opening and closing component. The above-mentioned opening and closing component can be an opening and closing component based on an arc motion of a hinge structure or a linear motion of a slide rail structure. For example, the above-mentioned target device can include but is not limited to: a refrigerator, a microwave oven, and a drawer cabinet. Correspondingly, the above-mentioned opening and closing component can include but is not limited to: a refrigerator door, a microwave oven door, and a drawer. The above-mentioned initial motion trajectory information can be a three-dimensional position information sequence representing the motion trajectory of the second target point during the opening or closing process of the opening and closing component. The above-mentioned second target point can be a point in a preset area (the area where the door handle is located) in the above-mentioned opening and closing component. For example, the above-mentioned second target point can be a point on the door handle of a refrigerator. The above-mentioned execution subject can be a computing device. The above-mentioned computing device can control the above-mentioned preset mechanical arm to open or close the opening and closing component in the target device. As an example, when the above-mentioned preset mechanical arm is used to automatically open the opening and closing components of the household appliance (target device), the above-mentioned execution subject can control the preset mechanical arm to open or close the opening and closing components in the household appliance (target device). When the above-mentioned preset mechanical arm is used to open and close the opening and closing components that need to be opened and closed in the automated production line, the above-mentioned execution subject can control the above-mentioned preset mechanical arm to automatically open or close the opening and closing components in the automated production line. For example, during the installation process of the door of an automobile production line, the door is installed on the automobile body (target device), and this process includes opening or closing the door so that it is properly aligned and fixed. The above-mentioned execution subject can control the above-mentioned preset mechanical arm to open or close the door (opening and closing component). The above-mentioned execution subject sends the control signal to the driver and actuator of the preset mechanical arm through a cable (such as USB, serial interface). The above-mentioned execution subject can also use a network or other communication methods to send instructions to the above-mentioned preset mechanical arm.

[0026] It should be noted that the above-mentioned wireless connection methods may include but are not limited to 3G / 4G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0027] In some optional implementations of some embodiments, the initial motion trajectory information is generated by the following steps:

[0028] The first step is to obtain pixel color information of two target points and a reference image of the target device in three-dimensional space from a preset database. The pixel color information may represent the pixel value of the pixel. For example, the pixel color information may be (255, 0, 0), indicating that the pixel of the second target point is pure red. The reference image may be an image of the target device at a preset shooting angle.

[0029] In the second step, the pixel color information of the second target point is determined as the reference pixel color information.

[0030] The third step is to obtain a video frame sequence of the opening and closing component of the target device during the opening or closing process from the preset database. The video frame sequence may represent a video of the opening and closing component of the target device during the opening or closing process. The video frame sequence may be a video frame sequence captured by a high frame rate video camera.

[0031] In the fourth step, according to the first video frame in the video frame sequence, perform the following generation steps:

[0032] In the first sub-step, the color information of each pixel point corresponding to each pixel point in the video frame is determined as the color information of each pixel point to be compared.

[0033] In the second sub-step, the similarity between each pixel color information to be compared and the reference pixel color information is determined as the screening similarity, wherein the screening similarity may be the cosine similarity between the vector corresponding to the pixel color information to be compared in the color space and the vector corresponding to the reference pixel color information in the color space.

[0034] The third sub-step is to determine the screening similarity that satisfies the preset screening condition among the determined screening similarities as the target screening similarity. The preset screening condition may be that the numerical value of the screening similarity is the maximum.

[0035] In the fourth sub-step, the color information of the pixel to be compared corresponding to the target screening similarity is determined as the target pixel color information.

[0036] In the fifth sub-step, the pixel corresponding to the color information of the target pixel is determined as the screening pixel.

[0037] In a sixth sub-step, the pixel position information of the screened pixel in the video frame is determined as the target pixel position information.

[0038] The seventh sub-step is to perform three-dimensional position information prediction processing on the target pixel position information based on the reference image and the video frame to generate the three-dimensional position information corresponding to the target pixel position information in the three-dimensional space. In practice, the above can be based on the reference image and the video frame through stereoscopic vision technology, and the target pixel position information is three-dimensionally reconstructed to obtain the three-dimensional position information corresponding to the target pixel position information in the three-dimensional space. For example, the three-dimensional position information can be (x, y, z, rx, ry, rz). The above x represents the horizontal coordinate value (usually refers to the east / west direction). The above y represents the vertical coordinate value (usually refers to the up / down direction). The above z represents the depth coordinate value (usually refers to the front / back direction). The above rx can be a rotation angle around the X axis (can be a pitch angle). The above ry can represent a rotation angle around the Y axis (can be a yaw angle). The above rz can represent a rotation angle around the Z axis (can be a roll angle).

[0039] In an eighth sub-step, the video frame is deleted from the video frame sequence to update the video frame sequence.

[0040] A ninth sub-step, in response to determining that the updated video frame sequence is not empty, executing the generating step again according to the first video frame in the updated video frame sequence.

[0041] In step 5, in response to determining that the updated video frame sequence is empty, the generated three-dimensional position information is arranged in the order in which the three-dimensional position information is generated to obtain a three-dimensional position information sequence, wherein each three-dimensional position information in the three-dimensional position information sequence corresponds to one of the screening pixel points.

[0042] The sixth step is to determine the above three-dimensional position information sequence as the initial motion trajectory information.

[0043] The above technical solution and its related contents, as an inventive point of an embodiment of the present disclosure, solve the second technical problem mentioned in the background technology, "Using laser scanning technology to collect the motion trajectory information of the opening and closing components during the opening or closing movement, since laser scanning requires time to scan point by point, the fast-moving opening and closing components may have moved during the scanning process, which may cause data loss or a large degree of distortion, resulting in low accuracy of the collected motion trajectory information". The factors that lead to low accuracy of the collected motion trajectory information are often as follows: Using laser scanning technology to collect the motion trajectory information of the opening and closing components during the opening or closing movement, since laser scanning requires time to scan point by point, the fast-moving opening and closing components may have moved during the scanning process, which may cause data loss or a large degree of distortion, resulting in low accuracy of the collected motion trajectory information. If the above factors are solved, the effect of improving the accuracy of the collected motion trajectory information can be achieved. In order to achieve this effect, first, the pixel color information of the second target point and the reference image of the above target device in three-dimensional space are obtained from a preset database. Thus, the pixel color information of the second target point used to generate the screening similarity can be obtained. The reference image for three-dimensional position information prediction processing can be obtained. Then, the pixel color information of the above second target point is determined as the reference pixel color information. Afterwards, a video frame sequence of the opening and closing component in the target device during the opening or closing process is obtained from the preset database. Thus, a video frame sequence representing the opening or closing process of the opening and closing component in the target device can be obtained. Then, according to the first video frame in the video frame sequence, the following generation steps are performed: a first sub-step, determining the color information of each pixel point corresponding to each pixel point in the video frame as the color information of each pixel point to be compared. Thus, the color information of each pixel point to be compared for determining each screening similarity can be obtained. A second sub-step, determining the similarity between each pixel point color information to be compared and the reference pixel point color information in each pixel point color information to be compared as the screening similarity. A third sub-step, determining the screening similarity that meets the preset screening condition among the determined screening similarities as the target screening similarity. Thus, the target screening similarity that represents the most similarity can be obtained. A fourth sub-step, determining the color information of the pixel point to be compared corresponding to the target screening similarity as the target pixel point color information. Thus, the target pixel point color information for determining the screening pixel point can be obtained. A fifth sub-step, determining the pixel point corresponding to the target pixel point color information as the screening pixel point. Thus, the screening pixel point can be determined. In the sixth sub-step, the pixel point position information of the screening pixel point in the video frame is determined as the target pixel point position information. Thus, the pixel point position information of the screening pixel point in the video frame can be obtained. In the seventh sub-step, based on the reference image and the video frame, the target pixel point position information is subjected to three-dimensional position information prediction processing to generate the three-dimensional position information corresponding to the target pixel point position information in the three-dimensional space.Thus, the three-dimensional position information corresponding to the target pixel point in the three-dimensional space can be generated. In the eighth sub-step, the video frame is deleted from the video frame sequence to update the video frame sequence. Thus, the video frame sequence can be updated. In the ninth sub-step, in response to determining that the updated video frame sequence is not empty, the generation step is performed again according to the first video frame in the updated video frame sequence. Next, in response to determining that the updated video frame sequence is empty, the generated three-dimensional position information is arranged in the generation order of the three-dimensional position information to obtain a three-dimensional position information sequence, wherein each three-dimensional position information in the three-dimensional position information sequence corresponds to one of the filtered pixel points. Thus, the three-dimensional position information sequence corresponding to the second target point in the process of opening or closing the opening and closing component can be obtained. Finally, the above three-dimensional position information sequence is determined as the initial motion trajectory information. Thus, the initial motion trajectory information can be obtained. Also because the video frame sequence representing the opening or closing process of the opening and closing component in the target device is used to generate the three-dimensional position information sequence corresponding to the second target point on the opening and closing component in the process of opening or closing the opening and closing component as the initial motion trajectory information. Using video frame sequences to generate initial motion trajectory information can capture the position of the second target point in the component at different time points in real time. In addition, the acquisition frequency of the video frame sequence is usually high-frequency and variable, which can reduce the degree of data loss or distortion during the rapid movement of the opening and closing components, and improve the accuracy of the acquired motion trajectory information.

[0044] Step 102: Based on the initial motion trajectory information, a preset motion trajectory prediction network is trained to obtain a trained motion trajectory prediction model.

[0045] In some embodiments, the execution subject may train the preset motion trajectory prediction network based on the initial motion trajectory information to obtain a trained motion trajectory prediction model. The preset motion trajectory prediction network may be a recurrent neural network (RNN) with the first position information and the second position information as input information and the mobile position information as output information. The first position information may indicate the location of the first target point. The first target point may be a preset point in a non-opening and closing component of the target device. The preset point may be a reference point fixed on the body of the target device (usually the side or top of the target device). As an example, in order to determine the opening angle of the refrigerator door and prevent the refrigerator door from opening by mistake (for example, the opening angle exceeds 180 degrees), a reference point fixed on the refrigerator body (usually the side or top of the refrigerator) is selected as the first target point. The second position information may indicate the location of the second target point. The second target point may be a preset point in an opening and closing component of the target device. For example, the second target point may be a point in a preset area (the area where the door handle is located) in the opening and closing component.

[0046] In some optional implementations of some embodiments, the execution subject may train a preset motion trajectory prediction network based on the initial motion trajectory information to obtain a trained motion trajectory prediction model through the following steps:

[0047] The first step is to determine a first preset value as the sliding window size. The first preset value may be 2.

[0048] The second step is to determine a second preset value as the sliding step length, wherein the second preset value may be 1.

[0049] The third step is to group the three-dimensional position information in the initial motion trajectory information based on the sliding window size and the sliding step size to obtain a three-dimensional position information group sequence. Each three-dimensional position information group in the three-dimensional position information group sequence includes the preceding three-dimensional position information and the succeeding three-dimensional position information. In practice, the execution subject can group the three-dimensional position information in the initial motion trajectory information according to the sliding window size and the sliding step size to obtain a three-dimensional position information group sequence. As an example, the three-dimensional position information can be {three-dimensional position information A, three-dimensional position information B, three-dimensional position information C}. The sliding window size can be 2. The sliding step size can be 1. According to the sliding window size of 2 and the sliding step size of 1, {three-dimensional position information A, three-dimensional position information B, three-dimensional position information C} are grouped to obtain a three-dimensional position information group sequence of {{three-dimensional position information A, three-dimensional position information B}, {three-dimensional position information B, three-dimensional position information C}}. Optionally, as an example, the above-mentioned three-dimensional coordinate position information may be {(x1, y1, z1, rx1, ry1, rz1), (x2, y2, z2, rx2, ry2, rz2), (x3, y3, z3, rx3, ry3, rz3)}. The above-mentioned three-dimensional position information group sequence may be {{(x1, y1, z1, rx1, ry1, rz1), (x2, y2, z2, rx2, ry2, rz2)}, {(x2, y2, z2, rx2, ry2, rz2), (x3, y3, z3, rx3, ry3, rz3)}}. The above-mentioned {(x1, y1, z1, rx1, ry1, rz1), (x2, y2, z2, rx2, ry2, rz2)} is a three-dimensional position information group. In the above {(x1, y1, z1, rx1, ry1, rz1), (x2, y2, z2, rx2, ry2, rz2)}, "(x1, y1, z1, rx1, ry1, rz1)" is the preceding three-dimensional position information of the three-dimensional position information group {(x1, y1, z1, rx1, ry1, rz1), (x2, y2, z2, rx2, ry2, rz2)}. The "(x2, y2, z2, rx2, ry2, rz2)" in the above {(x1, y1, z1, rx1, ry1, rz1), (x2, y2, z2, rx2, ry2, rz2)} is the subsequent three-dimensional position information of the three-dimensional position information group {(x1, y1, z1, rx1, ry1, rz1), (x2, y2, z2, rx2, ry2, rz2)}.

[0050] Step 4: for each three-dimensional position information group in the three-dimensional position information group sequence, perform the following steps:

[0051] In the first sub-step, the preceding three-dimensional position information in the three-dimensional position information group and the position information of the first target point are determined as sample information to be trained.

[0052] The second sub-step is to determine the subsequent three-dimensional coordinate information in the three-dimensional position information group as the sample subsequent three-dimensional coordinate information corresponding to the sample to-be-trained information.

[0053] In the third sub-step, the sample information to be trained and the subsequent three-dimensional coordinate information of the sample are determined as training samples.

[0054] The fifth step is to determine the determined training samples as a training sample set.

[0055] The sixth step is to train the preset motion trajectory prediction network based on the training sample set to obtain a trained motion trajectory prediction model.

[0056] In some optional implementations of some embodiments, the execution subject may train the preset motion trajectory prediction network based on the training sample set to obtain a trained motion trajectory prediction model through the following steps:

[0057] The first step is to select a preset number of training samples from the training sample set as the initial training sample set. In practice, the execution subject may randomly select a preset number of training samples from the training sample set as the initial training sample set.

[0058] In the second step, the following training steps are performed based on the initial training sample set:

[0059] In the first sub-step, each sample to-be-trained information of each initial training sample in the initial training sample set is input into a preset motion trajectory prediction network to obtain subsequent three-dimensional coordinate prediction information corresponding to each initial training sample in the above-mentioned initial training samples. The above-mentioned subsequent three-dimensional coordinate information can represent the position to be moved next by the opening and closing component predicted according to the initial training sample.

[0060] The second sub-step is to compare the sample post-order three-dimensional coordinate information in each of the above-mentioned initial training samples with the corresponding post-order three-dimensional coordinate prediction information. In practice, the above-mentioned execution subject can compare through a cross entropy loss function to determine the gap between the post-order three-dimensional coordinate prediction information corresponding to each of the at least one initial training sample and the corresponding sample post-order three-dimensional coordinate information. The above-mentioned gap can be represented by the loss function value.

[0061] The third sub-step is to determine whether the preset motion trajectory prediction network has achieved a preset optimization target based on the comparison result. The above optimization target may be that the loss function value is less than or equal to a preset value.

[0062] In a fourth sub-step, in response to determining that the preset motion trajectory prediction network achieves the above optimization goal, the preset motion trajectory prediction network is used as a trained motion trajectory prediction model.

[0063] The fifth sub-step is, in response to determining that the initial neural network does not achieve the above optimization goal, adjusting the network parameters of the initial neural network, and selecting a preset number of unused training samples from the above training sample set to form an initial training sample set, using the adjusted preset motion trajectory prediction network as the initial neural network, and performing the above training step again. As an example, the network parameters of the above preset motion trajectory prediction network can be adjusted using a back propagation algorithm (BP algorithm) and a gradient descent method (such as a small batch gradient descent algorithm).

[0064] Step 103 , in response to receiving preset request information indicating opening and closing movement of an opening and closing component in a target device, three-dimensional spatial position information of a first target point and three-dimensional spatial position information of a second target point in the target device are collected.

[0065] In some embodiments, the execution subject may, in response to receiving preset request information representing the opening and closing movement of the opening and closing component in the target device, collect the three-dimensional spatial position information of the first target point and the three-dimensional spatial position information of the second target point in the target device. The first target point is a point in a non-opening and closing component, and the second target point is a point in an opening and closing component. The preset request information representing the opening and closing movement of the opening and closing component in the target device may be text information for opening or closing the opening and closing component in the target device.

[0066] In some optional implementations of some embodiments, the three-dimensional spatial position information of the first target point and the three-dimensional spatial position information of the second target point are collected by the following steps:

[0067] The first step is to collect an image of the target device as the target device image. In practice, the execution subject may collect an image of the target device as the target device image by controlling an image collection device (eg, a camera).

[0068] The second step is to perform feature point extraction processing on the target device image to obtain first target point information and second target point information, wherein the first target point information includes first target point position information, and the second target point information includes second target point position information.

[0069] The third step is to generate the three-dimensional spatial position information corresponding to the first target point based on the first target point position information. In practice, first, the execution subject can obtain the image of the target device in the three-dimensional space from the preset database as a reference image. Then, based on the target device image and the reference image, the execution subject can perform three-dimensional reconstruction on the first target point position information through a three-dimensional reconstruction algorithm to obtain the position information corresponding to the first target point position information in the three-dimensional space as the three-dimensional spatial position information.

[0070] The fourth step is to generate the three-dimensional spatial position information corresponding to the second target point based on the second target point position information. In practice, the execution subject can perform three-dimensional reconstruction on the first target point position information based on the target device image and the reference image through a three-dimensional reconstruction algorithm to obtain the position information corresponding to the second target point position information in the three-dimensional space as the three-dimensional spatial position information.

[0071] In some optional implementations of some embodiments, the execution subject may perform feature point extraction processing on the target device image to obtain first target point information and second target point information through the following steps:

[0072] The first step is to convert the color information of each pixel of the target device image in the target device image into a pixel color feature vector. The color information may include RGB pixel value information and HSV (HSV color model) information. The RGB pixel value information may represent the pixel value of the pixel. The HSV (HSV color model) information may represent the hue, saturation and value of the pixel. The pixel color feature vector may be a vector representing the color information of the pixel of the target device image.

[0073] In the second step, the converted color feature vectors of each pixel are determined as a set of pixel color feature vectors.

[0074] The third step is to obtain the first preset color feature vector and the second preset color feature vector from a preset database, wherein the preset database may be a MySQL database.

[0075] In the fourth step, the similarity between the first preset color feature vector and each pixel color feature vector in the pixel color feature vector set is determined as a first similarity, wherein the first similarity can be represented by cosine similarity.

[0076] Step 5: Determine the first similarities that meet the first preset screening condition among the determined first similarities as the first target similarity. The first preset screening condition may be that the value of the first similarity is the largest.

[0077] In a sixth step, the position information of the pixel point of the target device image corresponding to the first target similarity in the target device image is determined as the first target point position information.

[0078] The seventh step is to determine the first target point position information as the first target point information.

[0079] In the eighth step, the similarity between the second preset color feature vector and each pixel color feature vector in the pixel color feature vector set is determined as a second similarity, wherein the second similarity can be represented by cosine similarity.

[0080] In the ninth step, the second similarities satisfying the second preset screening condition among the determined second similarities are determined as the second target similarity. The second preset screening condition may be that the value of the second similarity is the largest.

[0081] In the tenth step, the position information of the pixel point of the target device image corresponding to the second target similarity in the target device image is determined as the second target point position information.

[0082] In the eleventh step, the second target point position information is determined as the second target point information.

[0083] Step 104 : Determine the collected three-dimensional spatial position information of the first target point and the collected three-dimensional spatial position information of the second target point as first position information and second position information, respectively.

[0084] In some embodiments, the execution entity may determine the collected three-dimensional spatial position information of the first target point and the collected three-dimensional spatial position information of the second target point as the first position information and the second position information, respectively.

[0085] Step 105: determine the collection time of the first location information as the first collection time.

[0086] In some embodiments, the execution entity may determine the collection time of the first location information as the first collection time.

[0087] Step 106, based on the first location information, the second location information and the first collection time, perform the following monitoring movement steps:

[0088] Step 1061: input the first position information and the second position information into a motion trajectory prediction model to obtain movement position information.

[0089] In some embodiments, the execution subject may input the first position information and the second position information into a motion trajectory prediction model to obtain movement position information, wherein the movement position information may represent the position in the three-dimensional space where the opening and closing component is to move next after the second target point, predicted based on the first position information and the second position information.

[0090] Step 1062, after a preset time of the first collection time, control the preset collection device to collect the position information of the first target point in the target device as the mobile verification position information.

[0091] In some embodiments, the execution subject may control a preset acquisition device to acquire the location information of the first target point in the target device as the mobile verification location information after a preset time of the first acquisition time. The preset acquisition device may be a camera that acquires images of the target device.

[0092] Step 1063, determining the collection time of the mobile verification location information as the collection time to be updated.

[0093] In some embodiments, the execution entity may determine the collection time of the mobile verification location information as the collection time to be updated.

[0094] Step 1064: Update the collection time to be updated to the first collection time, so as to update the first collection time.

[0095] In some embodiments, the execution entity may update the to-be-updated collection time to the first collection time, so as to update the first collection time.

[0096] Step 1065, in response to determining that the mobile verification position information is the same as the first position information, controlling the preset robotic arm to clamp the opening and closing component in the target device at the position corresponding to the second position information and controlling the preset robotic arm to move the opening and closing component to the position corresponding to the mobile position information.

[0097] In some embodiments, the execution subject may, in response to determining that the mobile verification position information is the same as the first position information, control the preset mechanical arm to clamp the opening and closing component in the target device at the position corresponding to the second position information and control the preset mechanical arm to move the opening and closing component to the position corresponding to the mobile position information. In practice, the execution subject may control the mechanical arm to align the second target point on the opening and closing component with the position corresponding to the mobile position information (i.e., move the opening and closing component to move the second target point on the opening and closing component to the position corresponding to the mobile position information).

[0098] Step 1066: Update the mobile location information to the second location information to update the second location information.

[0099] In some embodiments, the execution entity may update the mobile location information to the second location information to update the second location information.

[0100] Step 1067, based on the first location information, the updated second location information and the updated first collection time, the monitoring movement step is performed again.

[0101] In some embodiments, the execution entity may execute the monitoring movement step again based on the first location information, the updated second location information and the updated first collection time.

[0102] The above-mentioned embodiments of the present disclosure have the following beneficial effects: through the opening and closing component moving method based on the mechanical arm of some embodiments of the present disclosure, the number of failures in opening and closing the opening and closing component is reduced, and the service life of the mechanical arm is extended. Specifically, the reasons for the increase in the number of failures in opening and closing the opening and closing component and the short service life of the mechanical arm are: based on the moving path planned when the device is in a fixed position, the mechanical arm is used to open and close the opening and closing component in the device at the fixed position. When the relative position of the device at the fixed position and the mechanical arm changes (for example, the mechanical arm or the device at the fixed position is moved), the mechanical arm may not be able to open and close the opening and closing component according to the moving path planned when it is in the fixed position, thereby increasing the number of failures of the mechanical arm to open and close the opening and closing component. At the same time, when the relative position of the device at the fixed position and the mechanical arm changes, based on the moving path planned when the device is in the fixed position, the opening and closing component is opened and closed, which may cause the mechanical arm and the device to be damaged due to excessive force after relative movement, thereby resulting in a short service life of the mechanical arm. Based on this, some embodiments of the present disclosure disclose a method for moving an opening and closing component based on a mechanical arm. First, the initial motion trajectory information of the opening and closing component in the target device is obtained. Thus, the initial motion trajectory information for training the preset motion trajectory prediction network can be obtained. Then, based on the above initial motion trajectory information, the preset motion trajectory prediction network is trained to obtain a trained motion trajectory prediction model. Thus, the mobile position information corresponding to the position to be moved next by the opening and closing component can be obtained in real time point by point according to the three-dimensional spatial position information of the second target point in the opening and closing component. Afterwards, in response to receiving the preset request information representing the opening and closing movement of the opening and closing component in the above target device, the three-dimensional spatial position information of the first target point and the three-dimensional spatial position information of the second target point in the above target device are collected, wherein the above first target point is a point in the non-opening and closing component, and the above second target point is a point in the opening and closing component. Then, the collected three-dimensional spatial position information of the first target point and the collected three-dimensional spatial position information of the second target point are respectively determined as the first position information and the second position information. Thus, the first position information and the second position information used to generate the mobile position information can be obtained. Afterwards, the collection time of the first position information is determined as the first collection time. Thus, the first acquisition time for monitoring the moving step can be obtained. Then, based on the first position information, the second position information and the first acquisition time, the following monitoring moving step is performed: the first position information and the second position information are input into the motion trajectory prediction model to obtain the moving position information. Thus, the moving position information corresponding to the position to be moved can be predicted in real time to obtain the moving position information representing the position to be moved next by the opening and closing component. Afterwards, after the preset time of the first acquisition time, the preset acquisition device is controlled to collect the position information of the first target point in the target device as the moving verification position information.Then, the collection time of collecting the mobile verification position information is determined as the collection time to be updated. Then, the collection time to be updated is updated to the first collection time to update the first collection time. Thus, the first collection time can be updated. Then, in response to determining that the mobile verification position information is the same as the first position information, the preset mechanical arm is controlled to clamp the opening and closing component in the target device at the position corresponding to the second position information, and the preset mechanical arm is controlled to move the opening and closing component to the position corresponding to the mobile position information. Thus, the preset mechanical arm can be used to move the opening and closing component to the position predicted in real time to be moved next. Then, the mobile position information is updated to the second position information to update the second position information. Thus, the second position information can be updated. Then, based on the first position information, the updated second position information and the updated first collection time, the above-mentioned monitoring movement step is performed again. Also because the monitoring movement is used in the process of opening and closing the opening and closing component in the device, the mobile position information corresponding to the position to which the opening and closing component is to move next is predicted point by point in real time according to the first position information and the second position information collected in real time. The preset robotic arm is used to move the opening and closing component to the position where the opening and closing component will move next in real time, thereby reducing the number of opening and closing movement failures caused by opening and closing the opening and closing component according to the planned moving path when it is in a fixed position, and reducing the possibility of damage to the equipment or the robotic arm due to excessive force after relative movement between the robotic arm and the equipment during the opening and closing movement according to the planned moving path, thereby extending the service life of the robotic arm.

[0103] Further references Figure 2 , which shows a structural schematic diagram of an electronic device 200 suitable for implementing some embodiments of the present disclosure. Figure 2 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0104] like Figure 2 As shown, the electronic device 200 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 201, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 202 or a program loaded from a storage device 208 into a random access memory (RAM) 202. In the RAM 202, various programs and data required for the operation of the electronic device 200 are also stored. The processing device 201, the ROM 202, and the RAM 202 are connected to each other via a bus 204. An input / output (I / O) interface 205 is also connected to the bus 204.

[0105] Typically, the following devices may be connected to the I / O interface 205: input devices 206 including, for example, a camera, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 207 including, for example, a preset mechanical arm, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 208 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 209. The communication device 209 may allow the electronic device 200 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 2 The electronic device 200 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 2 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0106] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 209, or installed from the storage device 208, or installed from the ROM 202. When the computer program is executed by the processing device 201, the functions defined in the method of some embodiments of the present disclosure are executed.

[0107] It should be noted that the computer-readable medium recorded in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In some embodiments of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, device or device. In some embodiments of the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0108] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0109] The computer-readable medium may be included in the electronic device; or it may exist independently without being assembled into the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains the initial motion trajectory information of the opening and closing component in the target device; based on the initial motion trajectory information, trains the preset motion trajectory prediction network to obtain a trained motion trajectory prediction model; in response to receiving preset request information representing the opening and closing movement of the opening and closing component in the target device, collects the three-dimensional spatial position information of the first target point and the three-dimensional spatial position information of the second target point in the target device, wherein the first target point is a point in the non-opening and closing component, and the second target point is a point in the opening and closing component; determines the collected three-dimensional spatial position information of the first target point and the collected three-dimensional spatial position information of the second target point as the first position information and the second position information respectively; determines the collection time of the first position information as the first collection time; based on the first position information, the second position information and the first collection time, the three-dimensional spatial position information of the first target point and the second target point are collected. The following monitoring movement steps are performed at the collection time: the first position information and the second position information are input into the motion trajectory prediction model to obtain the mobile position information; after the preset time of the first collection time, the preset collection device is controlled to collect the position information of the first target point in the target device as the mobile verification position information; the collection time for collecting the mobile verification position information is determined as the collection time to be updated; the collection time to be updated is updated to the first collection time to update the first collection time; in response to determining that the mobile verification position information is the same as the first position information, the preset mechanical arm is controlled to clamp the opening and closing component in the target device at the position corresponding to the second position information and the above-mentioned preset mechanical arm is controlled to move the opening and closing component to the position corresponding to the mobile position information; the mobile position information is updated to the second position information to update the second position information; based on the first position information, the updated second position information and the updated first collection time, the above-mentioned monitoring movement steps are performed again.

[0110] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0111] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0112] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0113] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of technical features, but should also cover other technical solutions formed by any combination of technical features or their equivalent features without departing from the inventive concept. For example, a technical solution formed by replacing the features with (but not limited to) the technical features with similar functions disclosed in the embodiments of the present disclosure.

Claims

1. A method for moving an opening and closing component based on a mechanical arm, comprising: Obtaining initial motion trajectory information of the opening and closing components in the target device; Based on the initial motion trajectory information, a preset motion trajectory prediction network is trained to obtain a trained motion trajectory prediction model; In response to receiving preset request information representing opening and closing movement of an opening and closing component in the target device, collecting three-dimensional spatial position information of a first target point and a second target point in the target device, wherein the first target point is a point in a non-opening and closing component and the second target point is a point in an opening and closing component; Determine the collected three-dimensional spatial position information of the first target point and the collected three-dimensional spatial position information of the second target point as first position information and second position information respectively; Determine the collection time of the first location information as the first collection time; Based on the first location information, the second location information and the first collection time, the following monitoring movement steps are performed: Inputting the first position information and the second position information into a motion trajectory prediction model to obtain movement position information; After a preset time of the first collection time, controlling a preset collection device to collect position information of a first target point in the target device as mobile verification position information; Determine the collection time of the mobile verification location information as the collection time to be updated; The collection time to be updated is updated to the first collection time, so as to update the first collection time. After the collection time to be updated is updated to the first collection time, so as to update the first collection time, the method further includes: In response to determining that the mobile verification location information is different from the first location information, updating the mobile verification location information to the first location information to update the first location information; Controlling a preset acquisition device to acquire position information of a second target point in the target device; Determining the position information of the second target point as the second position information to be updated; Updating the second location information to be updated to the second location information, so as to update the second location information; Based on the updated first collection time, the updated first location information and the updated second location information, executing the monitoring movement step again; In response to determining that the mobile verification position information is the same as the first position information, controlling the preset mechanical arm to clamp the opening and closing component in the target device at the position corresponding to the second position information and controlling the preset mechanical arm to move the opening and closing component to the position corresponding to the mobile position information; Updating the mobile location information to the second location information to update the second location information; Based on the first position information, the updated second position information and the updated first collection time, the monitoring movement step is performed again.

2. The method according to claim 1, wherein: The initial motion trajectory information includes each three-dimensional position information; And the method of training a preset motion trajectory prediction network based on the initial motion trajectory information to obtain a trained motion trajectory prediction model comprises: Determine a first preset value as a sliding window size; Determine the second preset value as the sliding step length; Based on the sliding window size and the sliding step size, each three-dimensional position information in the initial motion trajectory information is grouped to obtain a three-dimensional position information group sequence, wherein each three-dimensional position information group in the three-dimensional position information group sequence includes preceding three-dimensional position information and succeeding three-dimensional position information; For each three-dimensional position information group in the three-dimensional position information group sequence, the following steps are performed: Determine the preceding three-dimensional position information and the position information of the first target point in the three-dimensional position information group as sample information to be trained; Determining the subsequent three-dimensional coordinate information in the three-dimensional position information group as the sample subsequent three-dimensional coordinate information corresponding to the sample to be trained information; Determine the sample to be trained information and the subsequent three-dimensional coordinate information of the sample as a training sample; Determine the determined training samples as a training sample set; Based on the training sample set, the preset motion trajectory prediction network is trained to obtain a trained motion trajectory prediction model.

3. The method according to claim 2, wherein: The step of training the preset motion trajectory prediction network based on the training sample set to obtain a trained motion trajectory prediction model includes: Selecting a preset number of training samples from the training sample set as an initial training sample set; The following training steps are performed based on the initial training sample set: Inputting each sample to-be-trained information of each initial training sample in the initial training sample set into a preset motion trajectory prediction network to obtain subsequent three-dimensional coordinate prediction information corresponding to each initial training sample in the initial training samples; Comparing the sample post-order three-dimensional coordinate information in each of the initial training samples with the corresponding post-order three-dimensional coordinate prediction information; Determine whether the preset motion trajectory prediction network reaches the preset optimization goal according to the comparison result; In response to determining that the preset motion trajectory prediction network achieves the optimization goal, using the preset motion trajectory prediction network as a trained motion trajectory prediction model; In response to determining that the initial neural network does not achieve the optimization goal, the network parameters of the initial neural network are adjusted, and a preset number of unused training samples are selected from the training sample set to form an initial training sample set, and the adjusted preset motion trajectory prediction network is used as the initial neural network to perform the training step again.

4. The method according to claim 1, wherein the three-dimensional spatial position information of the first target point and the three-dimensional spatial position information of the second target point are collected by the following steps: Acquiring an image of the target device as a target device image; Performing feature point extraction processing on the target device image to obtain first target point information and second target point information, wherein the first target point information includes first target point position information, and the second target point information includes second target point position information; Based on the first target point position information, generate three-dimensional spatial position information corresponding to the first target point; Based on the second target point position information, three-dimensional spatial position information corresponding to the second target point is generated.

5. The method according to claim 4, wherein: The performing feature point extraction processing on the target device image to obtain first target point information and second target point information includes: Converting the color information of each pixel of the target device image into a pixel color feature vector; Determine the converted color feature vectors of each pixel as a set of color feature vectors of the pixel; Acquire a first preset color feature vector and a second preset color feature vector from a preset database; Determine the similarity between the first preset color feature vector and each pixel color feature vector in the pixel color feature vector set as a first similarity; Determine the first similarities that meet the first preset screening condition among the determined first similarities as the first target similarity; Determine the position information of the pixel point of the target device image corresponding to the first target similarity in the target device image as the first target point position information; Determining the first target point position information as first target point information; Determine the similarity between the second preset color feature vector and each pixel color feature vector in the pixel color feature vector set as a second similarity; Determine the second similarities that meet the second preset screening condition among the determined second similarities as the second target similarity; Determine the position information of the pixel point of the target device image corresponding to the second target similarity in the target device image as the second target point position information; The second target point position information is determined as second target point information.

6. An electronic device used in a method for moving an opening and closing component based on a mechanical arm, comprising: Preset robotic arm; one or more processors; a storage device having one or more programs stored thereon; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 5.

7. A computer readable medium having a computer program stored thereon, wherein: When the program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images