Control method, device and equipment and computer readable storage medium

Abstract

The invention provides a control method, device and equipment and a computer readable storage medium. The method comprises the steps that a first image of a target object in a folded state is acquired, the target object comprises N mark points, and N is a positive integer; performing coding processing on the first image to obtain a first coding result; on the basis of the first coding result, predicting first movement vectors of the N mark points when the target object is unfolded, and performing mapping processing on the N first movement vectors to obtain first control information of the electronic equipment; and based on the first control information, controlling the electronic equipment to carry out unfolding operation on the target object. According to the invention, the flattening precision and success rate can be improved.

Description

Technical Field

[0001] The present application relates to the field of automation control, and in particular to a control method, device, equipment and computer-readable storage medium. Background Art

[0002] With the development of industrial automation and intelligence, the robot arm can perform precise operations through the robot arm control system. For example, the robot arm can restore the folded or wrinkled cloth to a flat state or an unfolded state. This process involves placing the cloth on a workbench or other surface, and then using the robot arm to grab, unfold, release, etc., so that the shape changes such as wrinkles, folds, and bends of the cloth are corrected, and finally the cloth is unfolded. Using a robot arm to unfold the cloth can improve production efficiency, accuracy, and consistency.

[0003] In the related art, there are problems of unfolding failure or low unfolding efficiency in the unfolding process of fabric. Summary of the invention

[0004] The embodiments of the present application provide a control method, an apparatus, a device, and a computer-readable storage medium, which can improve the accuracy and success rate of flattening.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] The present application provides a control method, the method comprising:

[0007] Acquire a first image of the target object in a folded state, wherein the target object includes N marking points, where N is a positive integer;

[0008] Performing encoding processing on the first image to obtain a first encoding result;

[0009] Predicting the first motion vectors of the N marking points when performing an expansion operation on the target object based on the first encoding result, mapping the N first motion vectors to obtain first control information of the electronic device;

[0010] The electronic device is controlled to perform an expansion operation on the target object based on the first control information.

[0011] The present application provides a control device, the device comprising:

[0012] A first acquisition module, used to acquire a first image of the target object in a folded state, wherein the target object includes N marking points, where N is a positive integer;

[0013] A first encoding module, used to perform encoding processing on the first image to obtain a first encoding result;

[0014] A first prediction module, configured to predict, based on the first encoding result, first motion vectors of the N marking points when performing an expansion operation on the target object, and perform mapping processing on the N first motion vectors to obtain first control information of the electronic device;

[0015] The first control module is used to control the electronic device to perform an expansion operation on the target object based on the first control information.

[0016] An embodiment of the present application provides a computer device, including:

[0017] A memory for storing computer executable instructions;

[0018] The processor is used to implement the control method provided in the embodiment of the present application when executing the computer executable instructions stored in the memory.

[0019] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions for implementing the control method provided in the embodiment of the present application when the instructions are executed by a processor.

[0020] An embodiment of the present application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, the control method provided in the embodiment of the present application is implemented.

[0021] The embodiments of the present application have the following beneficial effects:

[0022] In an embodiment of the present application, when a target object in a folded state is flattened, the first image of the target object is first encoded to obtain a first encoding result; then, based on the first encoding result, the first motion vectors of the N marking points are predicted. Since the N marking points are included in the target object, and the first motion vector is used to characterize the movement of the marking points in the three-dimensional space, it is achieved by using the two-dimensional image to obtain N three-dimensional first motion vectors. In this way, the N marking points can be accurately guided to move in the three-dimensional space based on the three-dimensional first motion vector; then, the N first motion vectors are mapped to the first control information. Since the first motion vector belongs to three-dimensional information and the first motion vector is accurate, the first control information with high accuracy can be obtained through mapping, and the first control information also belongs to three-dimensional information. Based on this, the electronic device can realize accurate and efficient unfolding of the target object based on the first control information, thereby improving the success rate and efficiency of the unfolding. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 is a schematic diagram of a network architecture of a control system 100 provided in an embodiment of the present application;

[0024] Figure 2 is a schematic diagram of the structure of the terminal 400 provided in an embodiment of the present application;

[0025] Figure 3 It is a first flow chart of the control method provided in the embodiment of the present application;

[0026] Figure 4 is a schematic diagram of a process for predicting a first motion vector provided by an embodiment of the present application;

[0027] Figure 5 is a second flow chart of the control method provided in an embodiment of the present application;

[0028] Figure 6 This is a schematic diagram of a process for constructing sample labels provided in an embodiment of the present application;

[0029] Figure 7 This is a schematic diagram of a process for adjusting model parameters provided in an embodiment of the present application;

[0030] Figure 8 This is a schematic diagram showing a folding result of generating a data set provided in an embodiment of the present application;

[0031] Fig. 9 This is a schematic diagram showing the setting results of setting 3D points provided in an embodiment of the present application.

[0032] It should be pointed out that the above-mentioned "first" and "second" are only used to distinguish different solutions, and do not represent the degree of superiority or inferiority of the solutions or the priority in the implementation process. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0034] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0035] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0036] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0037] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0038] The relevant data collection and processing in the embodiments of this application should be strictly in accordance with the requirements of relevant laws and regulations when applied in examples, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of authorization of laws and regulations and the personal information subject.

[0039] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0040] 1) Fabric flattening refers to the process of restoring folded, curled, or wrinkled fabric to a flat or unfolded state through certain methods or techniques. This process can be achieved through mechanical equipment or automated systems. Fabric flattening can eliminate wrinkles, folds, or other deformations on the fabric surface, making it smooth and flat. This provides a standardized, flat fabric shape for subsequent operations (such as cutting, sewing, or packaging).

[0041] In clothing manufacturing scenarios, fabrics need to be flattened before cutting to ensure cutting accuracy and quality. In industrial production scenarios, some industrial fabrics need to be kept flat before use for subsequent processing or assembly. In logistics packaging scenarios, in order to facilitate fabric storage or transportation, fabrics need to be flattened first and then stacked or rolled up.

[0042] 2) Folded state refers to a specific form or structure formed by folding or stacking cloth, paper or other flexible materials under the action of some external force. Compared with the natural flat state of the material, this state includes characteristics such as wrinkles, bends or overlaps.

[0043] 3) The unfolded state, also known as the flat state, refers to the natural flat state of cloth, paper or other flexible materials without folds, wrinkles or overlaps. This state usually has a smooth and flat surface feature and is the state closest to the original shape of the material.

[0044] In industrial and manufacturing scenarios, fabric processing: In clothing production, fabrics need to be flat for precise cutting and sewing. In packaging and transportation scenarios, materials need to be flattened in some scenarios for storage or transportation. In automation and robotic operation scenarios, when automated systems need to handle flexible materials, they often flatten them from a folded state to facilitate subsequent operations, such as sewing, stacking or packaging. Flattening is the basis for fabric morphology modeling and operation planning. In scientific research scenarios, in the study of the physical or mechanical properties of flexible materials, the flat state is the standard state, which is used to measure material properties (such as thickness, tension, etc.).

[0045] The unfolded fabric is easier to cut, splice and other operations, avoiding errors caused by folding or wrinkles. The flat surface of the material can ensure the beauty and consistency of the product. In industrial production, the unfolded state facilitates the coordination of fabric and machine, improving processing efficiency. The flat state makes it easier to find defects on the surface of the material (such as holes and stains), and it is also convenient for mechanical equipment to accurately handle.

[0046] 4) Marking points refer to specific points evenly distributed on the surface of flexible materials, which are used to identify, track or reference the state and changes of flexible materials. Marking points can be set physically (such as markers, patches) or digitally (such as virtual points) for a variety of mission scenarios.

[0047] Marking points are clearly selected and identifiable locations on flexible materials, usually used to describe the shape, position, deformation or motion state of flexible materials. Marking points are unique, traceable and evenly distributed. Each marking point has a unique coordinate or number for easy distinction and identification; the position and state of the marking point can be tracked by vision, sensors or algorithms; the distribution of marking points is usually regular (such as a grid) or covers key areas according to needs.

[0048] The position and distribution of the markers can reflect the overall shape of the flexible material, such as whether the flexible material is flat or has wrinkles. The changes in the position and posture of the markers can be used to describe the deformation of the flexible material during operation or movement. When a robot or a mechanical arm operates a flexible material, the markers can be used as reference points for planning paths or performing operations. In 3D modeling, markers are used to define key nodes on the surface of flexible materials to facilitate physical simulation or dynamic analysis.

[0049] 5) Policy Network: A policy network is a function that accepts the state of the environment (such as sensor data, images, or other inputs) and outputs an action (or a probability distribution over actions) to achieve some goal (such as optimizing a path, executing a task).

[0050] The policy network generates actions suitable for the current task based on the input state, such as how a robot grasps an object or unfolds a piece of cloth. In automation tasks, the policy network can act as a controller to adjust the output action through continuous feedback. Through training, the policy network can learn the optimal strategy and achieve efficient operation or navigation in complex environments. The policy network can handle different tasks in different scenarios, such as flattening and folding cloth in flexible object operations.

[0051] In order to better understand the control method provided in the embodiment of the present application, the control method in the related art and its existing shortcomings are first described.

[0052] In the related art, the automatic flattening algorithm mainly focuses on the folding of clothes, and directly extracts the feature points on the clothes to perform predetermined folding or unfolding. However, the wrinkles of fabrics in the clothing processing process are many and irregular, and they cannot be flattened directly using the feature points. In addition, the related algorithms only use pictures as input, and cannot accurately judge the flatness of the fabric and the state of the obscured part of the fabric.

[0053] In combination with the above-mentioned related technologies, it can be known that the disadvantages of the related technologies at least include the inability to unfold wrinkled or folded fabrics, low fabric unfolding success rate and low fabric unfolding efficiency.

[0054] The embodiments of the present application provide a control method, device, computer equipment and computer-readable storage medium, which can improve the accuracy and success rate of flattening. The following describes an exemplary application of the computer equipment provided by the embodiments of the present application. The computer equipment provided by the embodiments of the present application can be implemented as various types of terminals such as laptops, tablet computers, desktop computers, set-top boxes, smart phones, smart speakers, smart watches, smart TVs, unmanned vehicles, drones, smart robots, aircraft, and robotic arms, and can also be implemented as servers. The following describes an exemplary application when the computer equipment is implemented as a terminal.

[0055] See also Figure 1 , Figure 1 It is a network architecture diagram of the control system 100 provided in an embodiment of the present application. To support an object flattening application, the terminal 400 is connected to the server 200 via the network 300. The network 300 may be a wide area network or a local area network, or a combination of the two.

[0056] The terminal 400 may be an electronic device including a robotic arm, a robotic hand or an automated device, and the terminal 400 is used to receive a flattening instruction for a target object, obtain a first image of the target object in a folded state, and generate a flattening request based on the first image; the terminal 400 is used to send the flattening request to the server 200; the server 200 is used to determine N marking points of the target object based on the first image, and return the N marking points to the terminal 400; the terminal 400 is used to encode the first image to obtain a first encoding result; based on the first encoding result, the first movement vectors of the N marking points when performing an expansion operation on the target object are predicted, the N first movement vectors are mapped to obtain first control information of the electronic device; based on the first control information, the terminal 400 is controlled to perform an expansion operation on the target object.

[0057] In an embodiment of the present application, when the terminal 400 flattens the target object in a folded state, it first encodes the first image of the target object to obtain a first encoding result; then, based on the first encoding result, it predicts the first motion vectors of the N marking points. Since the N marking points are included in the target object, and the first motion vector is used to characterize the movement of the marking points in the three-dimensional space, it is achieved by using the two-dimensional image to obtain the N three-dimensional first motion vectors. In this way, based on the three-dimensional first motion vector, the N marking points can be accurately guided to move in the three-dimensional space; then, the N first motion vectors are mapped to the first control information. Since the first motion vector belongs to three-dimensional information and the first motion vector is accurate, the first control information with high accuracy can be obtained through mapping, and the first control information also belongs to three-dimensional information. Based on this, the electronic device can realize accurate and efficient unfolding of the target object based on the first control information, thereby improving the success rate and efficiency of the unfolding.

[0058] In some embodiments, the N marking points of the target object may also be determined by the terminal 400 based on the first image. Based on this, the terminal 400 may acquire the N marking points of the target object from itself.

[0059] In some embodiments, the server 200 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The terminal and the server may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.

[0060] See also Figure 2 , Figure 2 is a schematic diagram of the structure of the terminal 400 provided in an embodiment of the present application, Figure 2 The terminal 400 shown includes: at least one processor 410, a memory 450, at least one network interface 420 and a user interface 430. The various components in the terminal 400 are coupled together via a bus system 440. It is understood that the bus system 440 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 440 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 440 is not shown in FIG. Figure 2 Various buses are labeled as bus system 440 .

[0061] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0062] The user interface 430 includes one or more output devices 431 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0063] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices that are physically remote from the processor 410.

[0064] The memory 450 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.

[0065] In some embodiments, memory 450 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0066] Operating system 451, including system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0067] A network communication module 452, used to reach other electronic devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 include: Bluetooth, Wireless Compatibility Certification (WiFi), and Universal Serial Bus (USB), etc.;

[0068] a presentation module 453 for enabling presentation of information via one or more output devices 431 (e.g., display screen, speaker, etc.) associated with the user interface 430 (e.g., a user interface for operating peripherals and displaying content and information);

[0069] The input processing module 454 is used to detect one or more user inputs or interactions from one of the one or more input devices 432 and translate the detected inputs or interactions.

[0070] In some embodiments, the device provided in the embodiments of the present application can be implemented in software. Figure 2 The control device 455 stored in the memory 450 is shown, which can be software in the form of a program and a plug-in, etc., including the following software modules: a first acquisition module 4551, a first encoding module 4552, a first prediction module 4553 and a first control module 4554. These modules are logical, so they can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be explained below.

[0071] In other embodiments, the device provided in the embodiments of the present application can be implemented in hardware. As an example, the device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the control method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application specific integrated circuits (Application Specific Integrated Circuit, ASIC), DSP, programmable logic device (Programmable Logic Device, PLD), complex programmable logic device (Complex Programmable Logic Device, CPLD), field programmable gate array (Field-Programmable Gate Array, FPGA) or other electronic components.

[0072] In some embodiments, the terminal can implement the control method provided in the embodiment of the present application by running various computer executable instructions or computer programs. For example, computer executable instructions can be microprogram-level commands, machine instructions or software instructions. The computer program can be a native program or software module in the operating system; it can be a native application (APPlication, APP), that is, a program that needs to be installed in the operating system to run, such as a material flattening APP and a material preparation process APP; it can also be a small program that can be embedded in any APP, that is, a program that can be run only by downloading it to a browser environment. In short, the above-mentioned computer executable instructions can be instructions in any form, and the above-mentioned computer program can be an application, module or plug-in in any form.

[0073] The control method provided in the embodiment of the present application will be described in conjunction with the exemplary application and implementation of the terminal provided in the embodiment of the present application.

[0074] The control method provided by the embodiment of the present application is described below. As mentioned above, the computer device implementing the control method of the embodiment of the present application can be a terminal, a server, or a combination of the two. Therefore, the execution subject of each step will not be repeatedly described below.

[0075] It should be noted that in the examples of the control method below, the target object is cloth as an example. Those skilled in the art can apply the control method provided in the embodiment of the present application to the deployment of other flexible materials such as film materials, plastics, leather, paper, etc., based on their understanding of the following. The embodiment of the present application can also be applied to various scenarios, including but not limited to the textile and clothing industry, electronic product manufacturing, packaging industry, industrial automation, logistics and warehousing, medical equipment and health industry, home services, artificial intelligence, environmental protection and recycling, etc.

[0076] See also Figure 3 , Figure 3 is a first flow chart of the control method provided in the embodiment of the present application, which will be combined with Figure 3 The steps shown are used to illustrate the control method provided in the embodiment of the present application. Figure 3 The execution subject is the terminal.

[0077] In step S101, a first image of a target object in a folded state is acquired.

[0078] In the embodiment of the present application, the target object includes N marking points, where N is a positive integer. Among them, the marking points can be evenly set on the surface of the target object in advance. For example, taking the target object as a 10 cm square cloth, the marking points can be set at a spacing of 1 cm on the square cloth. Based on this, 121 marking points can be set on the square cloth. The marking points can also be set at a spacing of 2 cm on the square cloth. Based on this, 36 marking points can be set on the square cloth. The marking points can also be set at preset positions on the surface of the target object. Following the above example, the marking points can be set at the four vertices and the center point of the square cloth, that is, 5 marking points are set.

[0079] In some embodiments, the target object may be an object that supports folding and flattening, and the constituent material of the target object may be a flexible material. Exemplarily, the target object may be cloth, film, plastic, and the like.

[0080] In some embodiments, the folded state refers to a state or shape formed when the target object is folded or stacked under the action of an external force. In the folded state, the target object includes features such as wrinkles, bends, or overlaps.

[0081] In some embodiments, an image acquisition device is deployed on the electronic device, and the image acquisition device can be used to acquire a first image of the target object in a folded state. Exemplarily, the image acquisition device can be a digital camera, a three-dimensional scanner, a depth camera, a structured light sensor, an infrared sensor, etc.

[0082] In step S102, encoding is performed on the first image to obtain a first encoding result.

[0083] In some embodiments, the above encoding process may include a method based on traditional image processing methods and a method based on image coding model.

[0084] In some embodiments, the implementation process based on traditional image processing methods may include: first performing image preprocessing such as denoising, normalization / standardization, and grayscale conversion on the first image to obtain a preprocessing result; then using feature extraction methods such as edge detection, texture feature extraction, and shape feature extraction to perform feature extraction on the preprocessing result to obtain a feature extraction result; finally, encoding the feature extraction result through feature encoding and dimensional compression to obtain a first encoding result.

[0085] In some embodiments, the stability of subsequent encoding is ensured by removing noise from the first image; the pixel values ​​of the first image are normalized to a preset range through normalization / standardization to ensure a uniform scale; if the first image is a color image, the first image can also be converted into a grayscale image to simplify processing.

[0086] In some embodiments, the edges of the first image can be extracted by operators such as Canny and Sobel to achieve edge detection; the gray-level co-occurrence matrix or local binary pattern can be used to extract texture features; the Hu moment, Fourier transform, etc. can be used to extract shape features to achieve shape feature extraction; the feature extraction results can be converted into digital form by coding algorithms such as quantization coding and Huffman coding to achieve feature coding; principal component analysis or other dimensionality reduction techniques can be used to reduce the dimension of the feature vector to achieve dimensional compression.

[0087] In some embodiments, the implementation process of the image coding model-based method may include: inputting a first image into the image coding model, and the image coding model may output a first coding result. Exemplarily, the image coding model may be a convolutional neural network, an autoencoder, a variational autoencoder, an encoder model, a convolutional neural network combined with a transformer, etc.

[0088] In step S103, the first motion vectors of the N marking points are predicted based on the first encoding result when performing an expansion operation on the target object, and the N first motion vectors are mapped to obtain first control information of the electronic device.

[0089] In some embodiments, the implementation process of the above-mentioned "first movement vector of N marking points when performing an expansion operation on the target object based on the prediction of the first encoding result" can be: the marking points can be located based on the first encoding result through a deep learning model or a traditional computer vision algorithm (such as Hough transform, contour extraction, etc.); and then the motion vector of each marking point is determined through motion estimation, that is, the moving distance and moving direction from the initial position to the target position.

[0090] In some embodiments, the implementation process of the above-mentioned "mapping N first motion vectors to obtain the first control information of the electronic device" can be: first, map each first motion vector to the coordinate system of the electronic device to ensure that the electronic device can understand and execute these vectors. For example, the two-dimensional coordinates in the first image are mapped to the three-dimensional workspace of the robotic arm. Secondly, inverse kinematics (IK) calculations are required to correspond the first motion vector of each marker point to the specific action of the robotic arm (such as joint angle, posture adjustment). Then, combined with the electronic device control strategy (such as PID control, model predictive control, etc.), each motion vector is smoothed to generate a mapping result, which can be a control command to ensure the accuracy and stability of the execution process. Finally, the first control information is generated based on the mapping result. Exemplarily, the first control information may include motion instructions of the robotic arm (such as the target angle, speed, acceleration, etc. of each joint) or instructions of other execution devices.

[0091] In some embodiments, the first prediction model can be used to predict the first motion vector of N markers, wherein the first prediction model can be a deep neural network, a regression model, etc. For example, when the first prediction model is a regression model, the first prediction model can be a diffusion policy model. The second prediction model can be used to implement the above mapping process to obtain the first control information of the electronic device. Similar to the above first prediction model, the second prediction model can also be a diffusion policy model.

[0092] In some embodiments, the diffusion strategy model is a model that generates actions or strategies by simulating the diffusion process, and is applicable to complex high-dimensional continuous action spaces. The core idea is to transform strategy learning into a process of generating control instructions under certain conditions by introducing the diffusion process.

[0093] In some embodiments, the diffusion strategy model generally includes two parts: one is to extract the features of the input data through a deep learning network (such as CNN, Transformer, etc.), and the other is to generate actions or control strategies through a diffusion process. The diffusion strategy model optimizes the goal of the generation model by gradually introducing noise during the training process, so that the model can generate more reasonable actions or strategies in the generation task. During the test process, the diffusion strategy model generates the final control strategy through a reverse diffusion process. Based on this, the characteristics of the diffusion strategy model include: generative characteristics, flexibility and scalability, among which the generative characteristics refer to the fact that the diffusion strategy model is different from the traditional regression model. The diffusion strategy model not only provides a point estimated output, but also generates an output containing a possible distribution, which makes the diffusion strategy model more capable of generating strategies in high-dimensional space and suitable for processing multimodal or highly uncertain environments. Flexibility and scalability mean that the diffusion strategy model can adaptively generate strategies according to different input data, and improve the generalization ability of the strategy through the diffusion process in the learning process.

[0094] In some embodiments, see Figure 4 The "first movement vector of the N marking points when performing an expansion operation on the target object based on the first encoding result prediction" in the above step S103 can be implemented by the following steps S1031 to S1033, which are described in detail below.

[0095] In step S1031, the first encoding result is mapped to a three-dimensional space to obtain three-dimensional data of the target object.

[0096] In some embodiments, the first encoding result can be mapped to a three-dimensional space using a trained first prediction model to achieve mapping of image features to the three-dimensional space; then, the three-dimensional data of the target object is generated using a decoding layer of the trained first prediction model.

[0097] In some embodiments, the three-dimensional data of the target object includes the spatial coordinates, shape, and other geometric features of the target object, wherein the other geometric features may be structure, posture, size, and the like.

[0098] In step S1032, first position information of N marking points is obtained from the three-dimensional data.

[0099] In some embodiments, N marker points can be identified from the three-dimensional data. Since the marker points are set in advance, they can be used to track and represent the shape of the target object. Then, the position of each marker point in the three-dimensional space is extracted to obtain the first position information, which can be a three-dimensional coordinate.

[0100] In step S1033, for each marking point, a first movement vector of the marking point is determined based on the initial position information of the marking point and the first position information of the marking point.

[0101] In some embodiments, for each marker point, the initial position of the marker point refers to the position information of the marker point after the target object is expanded. Based on this, the difference between the initial position information and the first position information can be determined to obtain the first movement vector of the marker point. The first movement vector includes a magnitude and a direction, and the magnitude of the first movement vector can be determined by the Euclidean distance formula, and the direction of the first movement distance refers to the direction from the coordinate represented by the first position information to the coordinate represented by the initial position information.

[0102] Through the above steps S1031 to S1033, by mapping the first code to the three-dimensional space, the spatial relationship and geometric form of the target object can be captured more accurately. For flexible materials such as cloth, three-dimensional modeling can better reflect the shape and movement of the cloth. And based on the position tracking and first movement vector determination of the marker point, the position change of each marker point can be accurately controlled during the unfolding operation. As a key reference point, the marker point can help the system accurately evaluate the unfolding or deformation process of the target object, so as to better guide the robot arm or other execution equipment to perform precise operations. The extraction and displacement calculation of the marker point helps to adjust and correct the operation strategy in real time to ensure that the unfolding or adjustment of the target object achieves the predetermined effect. In addition, through localized marker point tracking, the problem can be converted from global calculation to local calculation, thereby reducing the computational complexity. Only the movement of the marker point needs to be paid attention to, which reduces the processing of all points of the entire target object and improves efficiency. By combining the initial position and the target position, the movement vector of each marker point can be accurately calculated, providing more accurate instructions. These movement vectors include not only position changes, but also information such as posture and direction, which helps to generate more detailed and diverse control strategies.

[0103] In some embodiments, the implementation process of "mapping N first motion vectors to obtain first control information of the electronic device" in the above step S103 can be: splicing N first motion vectors to obtain spliced ​​motion vectors; extracting features of the spliced ​​motion vectors to obtain vector features; obtaining a mapping function, and mapping the vector features using the mapping function to obtain first control information, wherein the mapping function is determined during the training process of the prediction network, and the prediction network is used to implement the control method.

[0104] In some embodiments, N first motion vectors may be spliced ​​in a preset order. For example, assuming that the N first motion vectors are A1, A2, A3, A4, and A5, the spliced ​​motion vector may be (A1, A2, A3, A4, A5). Then, the spliced ​​motion vector may be subjected to feature extraction through a second prediction model to extract higher-level features, that is, to obtain vector features. Feature extraction may include dimensionality reduction, pooling operations, etc. Feature extraction is used to capture the nonlinear relationship of the spliced ​​feature vector and generate a feature representation with expressiveness. The vector feature is a low-dimensional vector that can characterize the information of the spliced ​​motion vector.

[0105] In some embodiments, the prediction network includes a second prediction model, and the mapping function is learned during the training process of the second prediction model. The task of the second prediction model is to map the input features to corresponding control information, and the control information can be an operation instruction of the robot arm. The mapping function is learned based on the input and output relationship in the training data.

[0106] In some embodiments, the vector feature is mapped using a mapping function to generate the first control information. Exemplarily, the first control information may include a motion control instruction, an operation path, etc. of the robot arm.

[0107] Continue to see Figure 3 , and the description continues with step S103 above.

[0108] In step S104, the electronic device is controlled to perform an expansion operation on the target object based on the first control information.

[0109] In some embodiments, the first control information may include flattening operation instructions. For example, the first control information may include the first operating point, first moving distance, first moving direction, stretching force, grasping point, etc. of the robotic arm. The first control information can be parsed first to obtain the control instructions; and then the electronic device can be controlled to perform the unfolding operation based on the control instructions.

[0110] In some embodiments, when the first control information includes a first operating point, a first moving distance and a first moving direction for the target object, based on this, the implementation process of the above step S104 can be: controlling the robotic arm of the electronic device to perform a grabbing operation on the first operating point; controlling the robotic arm to move the first moving distance in the first moving direction; controlling the robotic arm to perform a release operation on the first operating point.

[0111] In some embodiments, the position information of the first operating point can be obtained first, and then a grabbing instruction can be generated based on the position information of the first operating point; then, based on the grabbing instruction, a robotic arm of the electronic device can be controlled to perform a grabbing operation on the first operating point; then, a moving instruction can be generated based on the first moving direction and the first moving distance, and the robotic arm can be controlled to move the first moving distance in the first moving direction based on the moving instruction; finally, based on a preset release instruction, the robotic arm can be controlled to perform a release operation on the first operating point, thereby completing an expansion operation.

[0112] Through the above steps S101 to S104, when the target object in a folded state is flattened, the first image of the target object is first encoded to obtain a first encoding result; then, based on the first encoding result, the first motion vectors of the N marking points are predicted. Since the N marking points are included in the target object, and the first motion vector is used to characterize the movement of the marking point in the three-dimensional space, it is achieved by using the two-dimensional image to obtain N three-dimensional first motion vectors, so that the N marking points can be accurately guided to move in the three-dimensional space based on the three-dimensional first motion vector; then, the N first motion vectors are mapped to the first control information. Since the first motion vector belongs to three-dimensional information and the first motion vector is accurate, the first control information with high accuracy can be obtained through mapping, and the first control information also belongs to three-dimensional information. Based on this, the electronic device can realize accurate and efficient unfolding of the target object based on the first control information, thereby improving the success rate and efficiency of the unfolding.

[0113] In some embodiments, the control method of steps S101 to S104 can be implemented by a trained prediction network. Figure 5 Before executing the above step S101, a trained prediction network can be obtained by the training method of the following steps S001 to S004, which is described in detail below.

[0114] In step S001, an initial prediction network, image samples of training objects and sample labels are obtained.

[0115] In some embodiments, the initial prediction network refers to an untrained network, the model parameters of the initial prediction network may be default initial values, the initial prediction network may be a general network, the initial prediction network may be a deep learning network, a convolutional neural network, or a recurrent neural network.

[0116] In some embodiments, the training object may be an object of the same class as the target object. For example, the training object and the target object may both be cloth. The sample labels correspond to the image samples, and both are used to train the initial prediction network. The training object may be similar in shape to the target object. For example, the training object and the target object may both be 10 cm square cloth.

[0117] In some embodiments, image samples and sample labels corresponding to the image samples can be acquired or determined in advance and stored in the storage space of the electronic device, and the image samples and sample labels can be directly acquired from the storage space. The electronic device can acquire image sample information in advance through an image acquisition device; the electronic device can construct sample labels by performing a folding operation on the training object and acquiring position information before and after the folding, and the sample labels can include motion vector labels and control information labels.

[0118] In some embodiments, the training object includes a second operating point and M marking points, where M is a positive integer, wherein the marking points in the training object are also set in advance, and the setting method of the marking points of the training object can be the same as the setting method of the marking points of the target object, for example, both setting the marking points at a spacing of 1 cm; the setting methods of the two can also be different, for example, the marking points of the training object are set at a spacing of 1 cm, while the marking points of the target object are set at a spacing of 2 cm. Before executing the above step S001, refer to Figure 6 , the sample label can be constructed by following steps S01 to S04, which are described in detail below.

[0119] In step S01, a first starting position of a second operating point and a second starting position of M marking points are obtained before a folding operation is performed on the training object based on the second operating point, as well as a second image of the training object, a first ending position of the second operating point and a second ending position of the M marking points after the folding operation is performed.

[0120] In some embodiments, the first starting position, the second starting position, the first ending position, and the second ending position are expressed in the same form, and can all be three-dimensional coordinates.

[0121] In some embodiments, the second operation point refers to the point where the robot arm acts on the training object when folding the training object. The second operation point may be a marked point, that is, the second operation is one of the marked points; the second operation point may not be a marked point, that is, the second operation point is a surface point other than the marked point in the training object.

[0122] In some embodiments, when folding the training object based on the folding operation, the image of the training object, the second operation point, and the position information of each marking point are obtained in real time until the folding operation is completed, and an image set and a position information set are obtained. The first starting position of the second operation point and the second starting positions of the M marking points before the folding operation is performed on the training object are obtained from the position information set, and the first ending position of the second operation point and the second ending position of the M marking points after the folding operation is performed are also obtained from the position information set; and the second image of the training object after the folding operation is performed is obtained from the image set.

[0123] In step S02 , a third movement vector is determined based on the first starting position and the first ending position.

[0124] In some embodiments, the direction from the first starting position to the first ending position can be determined as the second moving direction of the third moving vector; the distance between the first starting position and the first ending position can be determined as the second moving distance of the third moving vector. The distance between the first starting position and the first ending position can be determined by the Euclidean distance formula, the Manhattan distance formula, the Chebyshev distance formula, the Mahalanobis distance formula, etc.

[0125] Exemplarily, assuming that the first starting position is (x1, y1, z1) and the first ending position is (x2, y2, z2), the third movement vector may be (x2-x1, y2-y1, z2-z1).

[0126] In step S03, for each marking point, a fourth movement vector of the marking point is determined based on the second starting position of the marking point and the second ending position of the marking point.

[0127] In some embodiments, the implementation process of the above step S03 is similar to the implementation process of the above step S02, and therefore, the implementation process of step S03 may refer to the implementation process of the above step S02.

[0128] In step S04, the second image is determined as an image sample, and a sample label is constructed based on the second operating point, the third motion vector and M fourth motion vectors.

[0129] In some embodiments, on the one hand, the second image is determined as an image sample, and on the other hand, the third motion vector is reversely processed to obtain a fifth motion vector; the M fourth motion vectors are respectively reversely processed to obtain M sixth motion vectors; and the M sixth motion vectors are determined as motion vector labels, and the second operating point and the fifth motion vector are determined as control information labels; finally, the motion vector label and the control information label are determined as sample labels.

[0130] In some embodiments, the third moving vector includes a second moving distance and a second moving direction, and the fourth moving vector includes a third moving distance and the second moving direction. Since the second operating point and the marking point are both surface points in the training object, the moving directions of the second operating point and the marking point are the same after folding, compared with before folding, which are both the third moving directions. Based on this, the implementation process of the above step S04 can be: determining the opposite direction of the second moving direction as the third moving direction; constructing a fifth moving vector using the second moving distance and the third moving direction; constructing M sixth moving vectors using M third moving distances and the third moving directions; determining the M sixth moving vectors as moving vector labels, and determining the second operating point and the fifth moving vector as control information labels; determining the moving vector label and the control information label as sample labels.

[0131] In some embodiments, the opposite direction of the second moving direction refers to the negative vector of the second moving direction. For example, assuming that the second moving direction is (v x , v y , v z ), then the third moving direction is (-v x , -v y , -v z ), that is, each component of the second moving direction is inverted to obtain the third moving direction. Continuing with the above example, assuming that the second moving distance is d2 and the third moving direction is (-v x , -v y , -v z ), then the fifth moving vector can be (d2, -v x , -v y , -v z ).

[0132] In some embodiments, for the third moving distance of each marking point, the third moving distance can be recorded as d3. Based on this, the sixth moving vector can be (d3, -v x , -v y , -v z ).

[0133] In some embodiments, the M sixth motion vectors are determined as motion vector labels, that is, the motion vector label includes the sixth motion vectors of the M marking points. The second operation point and the fifth motion vector are determined as control information labels. Continuing with the above example, assuming that the second operation point is recorded as O2, the control information label can be expressed as (O2, d2, -v x , -v y , -v z ), the control information label indicates: based on the second operating point O2, along the third moving direction (-v x , -v y , -v z ) moves the second moving distance d2.

[0134] Through the above steps S01 to S04, by associating the second operating point and the fourth movement vector of the marking point with the image sample, accurate displacement information can be provided to the model. This helps the training model predict the precise folding operation path and the position of the marking point, thereby improving the prediction accuracy of the folding process of the cloth or the target object. In the process of constructing the sample label, not only the second image is used, but also the third movement vector and the fourth movement vector in the folding operation are combined. This combination of multimodal data helps the model understand the complex spatial relationship in the folding process, thereby reducing the model error. By collecting different operating points, marking points and their displacement information, a large number of sample label pairs are generated, which increases the diversity of the training set and helps the model adapt to various possible folding operation situations. This enables the model to have stronger generalization ability when facing different cloth forms and folding operations. In addition, no matter how the cloth is folded, the model can adapt to different initial and final states by learning the movement vectors of different marking points and operating points, and thus can predict more complex folding processes. And through this process, the model can automatically learn the mapping relationship from image to folding operation, reducing the need for manual intervention. Image samples and sample labels together constitute the input and label pairs. Through these data, the model learns how to predict and perform folding operations, thus improving the automation level of the system. By accurately tracking and calculating the displacement of each marker point and operation point, the folding error caused by inaccurate operation can be effectively reduced, ensuring a smoother and more stable folding process. The folding of fabric may cause uneven wrinkles or deformation. By combining the second image, the third motion vector, and the fourth motion vector, the model can make adaptive adjustments according to the specific state of the fabric during the folding process, so as to better cope with different forms of fabric.

[0135] Continue to see Figure 5 , and the description continues with step S001 above.

[0136] In step S002, the image sample is encoded using the initial prediction network to obtain a sample encoding result.

[0137] In some embodiments, the initial prediction network includes a coding model, a first prediction model and a second prediction model. The implementation process of the above step S002 may be: encoding the image sample using the coding model to obtain a sample encoding result.

[0138] In some embodiments, the image sample may be input into the coding model, so as to use the coding model to encode the image sample and obtain the sample coding result. The image coding model may be a convolutional neural network, an autoencoder, a variational autoencoder, an encoder model, a convolutional neural network combined with a transformer, etc. Exemplarily, the image coding model may be a third generation mobile network-small (MobileNetV3-Smal) model.

[0139] In step S003, an expansion prediction process is performed based on the sample encoding result to obtain second control information.

[0140] In some embodiments, the implementation process of the above step S003 may be: using the first prediction model to perform motion vector prediction on the sample encoding result to obtain a second motion vector; using the second prediction model to perform control information prediction on the second motion vector to obtain second control information.

[0141] In some embodiments, the sample encoding result can be input into the first prediction model, and the first prediction model can be used to output M second motion vectors; and the M second motion vectors can be input into the second prediction model, and the second prediction model can be used to output the second control information. Exemplarily, both the first prediction model and the second prediction model can be CNN-based Diffusion Policy models, and the model parameters of the two models can be different.

[0142] In step S004, based on the second control information and the sample labels, the model parameters of the initial prediction network are adjusted to obtain a trained prediction network.

[0143] In some embodiments, the model parameters of the first prediction model and the second prediction model can be adjusted respectively, so as to adjust the initial prediction network model parameters, thereby obtaining a prediction network after the model parameters are adjusted, that is, obtaining a trained prediction network.

[0144] In some embodiments, the sample labels include motion vector labels and control information labels, see Figure 7 The above step S004 can be implemented by the following steps S0041 to S0043, which are described in detail below.

[0145] In step S0041, a first loss value is determined based on the second motion vector and the motion vector label, and a model parameter of the first prediction model is adjusted based on the first loss value until a first training termination condition is reached, thereby obtaining a trained first prediction model.

[0146] In some embodiments, the first training termination condition may be that the first loss value is less than a loss threshold, wherein the loss threshold may be a value set according to experience, and illustratively, the loss threshold may be 0.05, 0.1, etc. The first training termination condition may also be that the number of training times reaches a number threshold, and the number threshold may be a value set according to experience, and illustratively, the number threshold may be 500, 1000, 1500, etc. The first training termination condition may also be that the first loss value is less than the loss threshold and the number of training times reaches the number threshold.

[0147] In some embodiments, the gradient of the first prediction model can be determined by a back propagation algorithm according to the first loss value, and the model parameters of the first prediction model can be adjusted or updated. The goal of back propagation is to minimize the loss function by adjusting the model parameters.

[0148] The goal of backpropagation is to adjust the model parameters to minimize the loss function. By calculating the gradient and updating the parameters, the model gradually "learns" how to better predict the true value, thereby reducing the loss. This process is iterative, and the model gradually improves its predictive ability through multiple iterations.

[0149] In some embodiments, the chain rule and the intermediate results (such as activation values) saved during the forward propagation process can be used to reversely calculate the gradient of each layer. Parameter update, using the calculated gradient, adjust the model parameters according to the update rule. Iterative optimization, repeat the above gradient calculation and parameter update process until the model converges or reaches a predetermined number of iterations. The back propagation algorithm can effectively optimize the model parameters to minimize the loss function, thereby improving the prediction accuracy.

[0150] In some embodiments, if the first training termination condition is not met, the first loss value continues to be determined and the model parameters are adjusted, and multiple rounds of training are performed until the first training termination condition is met, thereby obtaining a trained first prediction model.

[0151] In step S0042, a second loss value is determined based on the second control information and the control information tag, and model parameters of the second prediction model are adjusted based on the second loss value until a second training termination condition is reached, thereby obtaining a trained second prediction model.

[0152] In some embodiments, the implementation process of the above step S0042 is similar to the implementation process of the above step S0041, and therefore, the implementation process of the above step S0042 may refer to the implementation process of the above step S0041.

[0153] In step S0043, a trained prediction network is constructed using the encoding model, the trained first prediction model and the trained second prediction model.

[0154] In some embodiments, the encoding model, the trained first prediction model and the trained second prediction model may be cascaded to obtain a trained prediction network.

[0155] Through the above steps S0041 to S0043, by training two models (the first prediction model and the second prediction model) respectively, each model is specially optimized for different tasks (such as motion vectors and control information). This independent training method helps to reduce interference between models, so that each model can achieve higher accuracy on its focused tasks. By adjusting each prediction model based on the loss value of the motion vector label and the control information label, the performance of each model can be optimized more meticulously. This ensures that the output of each model is more in line with actual needs and improves the accuracy and reliability of the prediction. First, the first prediction model and the second prediction model are trained to optimize the network parameters for specific tasks respectively, and then these models are combined to finally obtain a trained complete prediction network. Such staged training can make the training process more efficient and avoid the interference or complexity that may be caused by multi-task training at the same stage. By optimizing the loss of each prediction model separately, over-adjustment or premature convergence of model parameters is avoided. The prediction network can be gradually adjusted at each stage to better find the global optimal solution and improve training efficiency. Combining the encoding model, the trained first prediction model, and the trained second prediction model to form a trained prediction network enables the system to understand the problem from multiple angles and levels, increasing the network's expressive power. The final trained prediction network can effectively handle the relationship between different tasks and improve its adaptability to unknown scenarios. By training the two prediction models separately, mutual interference can be reduced, ensuring that each model can fully learn in its task and avoid over-reliance on certain features. This allows each prediction model to better generalize to other situations or tasks.

[0156] Through the above steps S001 to S004, the relationship between the input image and the folding operation is automatically learned by training the prediction network, avoiding the problem of manually designing complex rules in traditional methods. The system can automatically extract features from the data and learn operation strategies, thereby improving the degree of automation. By optimizing and adjusting the network using sample labels, the network can continuously adjust its model parameters to improve prediction accuracy. This self-optimization capability enables the network to automatically adapt and improve the effect when facing different folding tasks. By training the prediction network based on image samples and sample labels, the dependence on manually labeled data can be reduced. Especially in complex folding operations, valuable information can be directly extracted from the image by training the network, without manually labeling each detail. Once the training is completed, the trained prediction network can automatically predict the first control information of the folding operation from the first image, further reducing the complexity of manual intervention and control decision-making. The system can automatically complete the operation without human intervention. The image samples and sample labels in the training process cover different types of folding operations. Through training with these data, the prediction network can learn how to perform accurate operation control in different scenarios. Diversified training data helps the network generalize better, and even in new scenarios, it can make effective predictions. By encoding the image samples, the prediction network can automatically extract features from the input image and generate the corresponding second control information. In this way, the image is directly converted into control instructions for the target operation, avoiding manual calculation and derivation process, and improving efficiency and accuracy. The trained prediction network can quickly generate the first control information and perform the folding operation, thereby improving the operation efficiency. Traditional manual or rule-driven folding methods are usually time-consuming, while the prediction network can generate reasonable operation instructions based on the image and control information in a short time, ensuring the efficient completion of the folding task.

[0157] In some embodiments, after the above-mentioned step S104, the following may also be performed: obtaining a third image of the target object; when the third image represents that the target object is in a folded state, encoding the third image to obtain a second encoding result; based on the second encoding result, predicting the seventh motion vectors of the N marking points when performing an unfolding operation on the target object, mapping the N seventh motion vectors to obtain third control information of the electronic device; and controlling the electronic device to perform an unfolding operation on the target object based on the third control information until the target object reaches the unfolded state.

[0158] In some embodiments, the implementation process of "obtaining the third image of the target object" is similar to the implementation process in the above step S101. Therefore, the implementation process of "obtaining the third image of the target object" can refer to the implementation process of the above step S101. If the third image represents that the target object is in an unfolded state or a flat state, the process ends. If the third image represents that the target object is in a folded state, "encode the third image to obtain a second encoding result; predict the seventh motion vector of the N marking points when the target object is unfolded based on the second encoding result, map the N seventh motion vectors to obtain the third control information of the electronic device; control the electronic device to unfold the target object based on the third control information."

[0159] In some embodiments, the above-mentioned "encoding the third image to obtain the second encoding result; based on the second encoding result, predicting the seventh motion vectors of the N marking points when performing the expansion operation on the target object, mapping the N seventh motion vectors to obtain the third control information of the electronic device; controlling the electronic device to perform the expansion operation on the target object based on the third control information" is similar to the implementation process of the above-mentioned steps S102 to S104. Therefore, the implementation process of the above-mentioned "encoding the third image to obtain the second encoding result; based on the second encoding result, predicting the seventh motion vectors of the N marking points when performing the expansion operation on the target object, mapping the N seventh motion vectors to obtain the third control information of the electronic device; controlling the electronic device to perform the expansion operation on the target object based on the third control information" can refer to the implementation process of the above-mentioned steps S102 to S104. In this way, through multiple and step-by-step expansion operations, accurate expansion of the target object is achieved.

[0160] The following is an explanation of an exemplary application of the embodiments of the present application in a practical application scenario.

[0161] The embodiment of the present application proposes a control method, which realizes the flattening of cloth based on 3D (Dimension, D) point tracking, simulates the process of random folding of cloth by a robot arm in a simulation environment, and collects 3D point tracking information, cloth pictures at multiple moments and robot arm movements of the cloth in the folding process through a deformable geometric scene reconstruction (DeformGS) algorithm, which is used to train a diffusion strategy model 1 based on a convolutional neural network and a diffusion strategy model 2 based on a convolutional neural network, and the 3D point tracking information includes 3D point position information and direction information. In practical applications, the wrinkled cloth picture at the current moment is input to the strategy network, and a series of robot arm movements are output, so as to control the robot arm to gradually flatten the wrinkled cloth. Among them, the cloth corresponds to the target object or training object in other embodiments, the 3D point corresponds to the marking point in other embodiments, the cloth picture corresponds to the image sample in other embodiments, the diffusion strategy model 1 based on the convolutional neural network corresponds to the first prediction model in other embodiments, the diffusion strategy model 2 based on the convolutional neural network corresponds to the second prediction model in other embodiments, the wrinkled cloth picture corresponds to the first image in other embodiments, and the robot arm movement corresponds to the unfolding operation in other embodiments.

[0162] In order to solve the problem of flattening wrinkled and irregular fabrics by a robot arm during garment processing, the embodiment of the present application collects 3D point tracking information, fabric images at multiple times, and robot arm movements during the folding process, and trains a strategy network based on the 3D point tracking information, fabric images at multiple times, and robot arm movements during the folding process, so that the robot arm can gradually flatten the wrinkled fabric based on the trained strategy network in practice. The strategy network corresponds to the prediction network in other embodiments.

[0163] The control method of the embodiment of the present application includes establishing a data set and training a policy network, wherein the data set corresponds to image samples and sample labels of the training object in other embodiments.

[0164] To establish the data set, in a simulation environment, a robot arm is used to perform random folding operations on the cloth, and the operation vector, the red, green, and blue (RGB) image of the cloth, and the 3D point tracking vector on the cloth are recorded. The operation vector includes the position, moving distance, and direction of the operation point. The cloth starts from a flat state, is folded multiple times, and the operation vector, the RGB image of the cloth, and the 3D point tracking vector on the cloth are recorded. The 3D point tracking vector includes the change distance and change direction of the 3D tracking point, such as Figure 8 As shown in the figure, the process of gradually generating data is Figure 8801 is an operation point, and the arrowed line segment 802 is a change distance and a change direction of the operation point. The operation point corresponds to the second operation point in other embodiments, the flat state corresponds to the unfolded state in other embodiments, the cloth RGB image corresponds to the image sample in other embodiments, and the 3D point tracking vector corresponds to the fourth movement vector in other embodiments.

[0165] The 3D point change is calculated by the DeformGS algorithm, and the 3D point change refers to the 3D point tracking vector. The DeformGS algorithm can achieve high-quality dynamic scene reconstruction and high-precision 3D tracking for highly deformed cloth objects with occlusion and shadows. Fig. 9 As shown, 3D points or 3D tracking points 901 are evenly set on the cloth, and the change amount of the 3D tracking points is recorded after each robot arm operation is completed, such as Fig. 9 In the line segment 902 with an arrow, the direction of the arrow indicates the change direction, and the length of the line segment indicates the change distance.

[0166] For the training of the policy network, the policy network of the embodiment of the present application includes a third-generation mobile network-small model for image encoding, a Diffusion Policy model 1 for establishing RGB images and 3D point tracking vectors, and a Diffusion Policy model 2 for establishing 3D point tracking vectors and robotic arm operation vectors. The Diffusion Policy model has strong generalization performance. Among them, the third-generation mobile network-small model for image encoding corresponds to the encoding model in other embodiments.

[0167] The process of generating folding data is reversed, which is the process of flattening the cloth. The operation vector in the data set is reversed to the action of the robot arm flattening the cloth, and the 3D point tracking vector is reversed. Finally, the reversed operation vector and the reversed 3D tracking point vector are used to train the policy network.

[0168] Each RGB image is encoded using the pre-trained MobileNetV3-Smal model, and the image at time t is used as the input of Diffusion Policy Model 1, and the change in the 3D tracking point at the corresponding time is used as the sample label of Diffusion Policy Model 1. The data of 3D point tracking vectors at three consecutive times can be used as a sequence, such as (t+2, t+1, t) as the input of Diffusion Policy Model 2, and the corresponding output is 3 consecutive actions of the robot arm (t, t-1, t-2).

[0169] Compared to the method of operating by inputting pictures and outputting operations, the embodiment of the present application adds 3D point tracking information, so that the strategy network can comprehensively consider the state of the entire piece of cloth, so that the robot arm can flatten the cloth more accurately.

[0170] It can be understood that in the embodiments of the present application, the collection, use and processing of data related to the first image, image samples, initial prediction network, first starting position, second starting position, second image, first ending position, second ending position, third image, etc. need to comply with relevant laws, regulations and standards.

[0171] The following is a description of an exemplary structure of the control device 455 provided in the embodiment of the present application being implemented as a software module. In some embodiments, Figure 2 As shown, the software modules stored in the control device 455 of the memory 450 may include:

[0172] The first acquisition module 4551 is used to acquire a first image of the target object in a folded state, wherein the target object includes N marking points, and N is a positive integer; the first encoding module 4552 is used to encode the first image to obtain a first encoding result; the first prediction module 4553 is used to predict, based on the first encoding result, first movement vectors of the N marking points when the target object is unfolded, and map the N first movement vectors to obtain first control information of the electronic device; the first control module 4554 is used to control the electronic device to unfold the target object based on the first control information.

[0173] In some embodiments, the first prediction module 4553 is further configured to:

[0174] Map the first encoding result to a three-dimensional space to obtain three-dimensional data of the target object; obtain first position information of N marking points from the three-dimensional data; and for each marking point, determine a first movement vector of the marking point based on the initial position information of the marking point and the first position information of the marking point.

[0175] In some embodiments, the first prediction module 4553 is further configured to:

[0176] Splicing N of the first motion vectors to obtain a spliced ​​motion vector; performing feature extraction on the spliced ​​motion vector to obtain a vector feature; obtaining a mapping function, and using the mapping function to map the vector feature to obtain the first control information, wherein the mapping function is determined during the training process of a prediction network, and the prediction network is used to implement the control method.

[0177] In some embodiments, the first control information includes a first operation point, a first moving distance, and a first moving direction for the target object, and the first control module 4554 is further used to:

[0178] Control the mechanical arm of the electronic device to perform a grabbing operation on the first operating point; control the mechanical arm to move the first moving distance in the first moving direction; and control the mechanical arm to perform a releasing operation on the first operating point.

[0179] In some embodiments, the control method is implemented by a trained prediction network, and the software module stored in the control device 455 of the memory 450 further includes:

[0180] The second acquisition module is used to obtain the initial prediction network, the image samples of the training object and the sample labels; the second encoding module is used to encode the image samples using the initial prediction network to obtain the sample encoding results; the second prediction module is used to carry out prediction processing based on the sample encoding results to obtain the second control information; the first adjustment module is used to adjust the model parameters of the initial prediction network based on the second control information and the sample labels to obtain the trained prediction network.

[0181] In some embodiments, the initial prediction network includes a coding model, a first prediction model and a second prediction model, and the second coding module is further used to: use the coding model to encode the image sample to obtain the sample encoding result; the second prediction module is further used to: use the first prediction model to perform motion vector prediction on the sample encoding result to obtain a second motion vector; use the second prediction model to perform control information prediction on the second motion vector to obtain second control information.

[0182] In some embodiments, the sample label includes a motion vector label and a control information label, and the first adjustment module is also used to: determine a first loss value based on the second motion vector and the motion vector label, and adjust the model parameters of the first prediction model based on the first loss value until the first training termination condition is reached to obtain the trained first prediction model; determine a second loss value based on the second control information and the control information label, and adjust the model parameters of the second prediction model based on the second loss value until the second training termination condition is reached to obtain the trained second prediction model; use the encoding model, the trained first prediction model and the trained second prediction model to construct the trained prediction network.

[0183] In some embodiments, the training object includes a second operating point and M marking points, where M is a positive integer, and the software module stored in the control device 455 of the memory 450 further includes:

[0184] a third acquisition module, configured to acquire a first starting position of the second operating point and a second starting position of the M marking points before a folding operation is performed on the training object based on the second operating point, as well as a second image of the training object, a first ending position of the second operating point and a second ending position of the M marking points after the folding operation is performed; a first determination module, configured to determine a third motion vector based on the first starting position and the first ending position; a second determination module, configured to determine, for each marking point, a fourth motion vector of the marking point based on the second starting position and the second ending position of the marking point; and a first construction module, configured to determine the second image as the image sample, and to construct the sample label based on the second operating point, the third motion vector and the M fourth motion vectors.

[0185] In some embodiments, the third moving vector includes a second moving distance and a second moving direction, the fourth moving vector includes a third moving distance and the second moving direction, and the construction module is further used to: determine the opposite direction of the second moving direction as the third moving direction; construct a fifth moving vector using the second moving distance and the third moving direction; construct M sixth moving vectors using M of the third moving distances and the third moving directions; determine the M sixth moving vectors as moving vector labels, and determine the second operating point and the fifth moving vector as control information labels; determine the moving vector label and the control information label as the sample label.

[0186] In some embodiments, the software modules stored in the control device 455 of the memory 450 further include:

[0187] A fourth acquisition module is used to acquire a third image of the target object; a third encoding module is used to encode the third image to obtain a second encoding result when the third image represents that the target object is in a folded state; a third prediction module is used to predict the seventh motion vectors of the N marking points when performing an unfolding operation on the target object based on the second encoding result, and map the N seventh motion vectors to obtain third control information of the electronic device; a second control module is used to control the electronic device to perform an unfolding operation on the target object based on the third control information until the target object reaches the unfolded state.

[0188] The embodiment of the present application provides a computer program product or a computer program, which includes computer executable instructions, and the computer executable instructions are stored in a computer readable storage medium. The processor of the computer device reads the computer executable instructions from the computer readable storage medium, and the processor executes the computer executable instructions, so that the computer device executes the control method described above in the embodiment of the present application.

[0189] The embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, the processor will be caused to execute the control method provided by the embodiment of the present application, for example, Figure 3 , 5 The control method is shown.

[0190] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or it may be various devices including one or any combination of the above memories.

[0191] In some embodiments, computer executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0192] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0193] By way of example, computer executable instructions may be deployed to be executed on one computing device or on multiple computing devices located at one site or on multiple computing devices distributed across multiple sites and interconnected by a communication network.

[0194] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A control method, characterized in that: The method comprises: Acquire a first image of the target object in a folded state, wherein the target object includes N marking points, where N is a positive integer; Performing encoding processing on the first image to obtain a first encoding result; Predicting the first motion vectors of the N marking points when performing an expansion operation on the target object based on the first encoding result, mapping the N first motion vectors to obtain first control information of the electronic device; The electronic device is controlled to perform an expansion operation on the target object based on the first control information.

2. The method according to claim 1, characterized in that When performing an expansion operation on the target object based on the prediction of the first encoding result, the first motion vectors of the N marking points include: Mapping the first encoding result to a three-dimensional space to obtain three-dimensional data of the target object; Acquire first position information of the N marking points from the three-dimensional data; For each of the marking points, a first movement vector of the marking point is determined based on the initial position information of the marking point and the first position information of the marking point.

3. The method according to claim 1, characterized in that The mapping process is performed on the N first motion vectors to obtain first control information of the electronic device, including: splicing N of the first motion vectors to obtain a spliced ​​motion vector; Extracting features from the spliced ​​motion vector to obtain vector features; A mapping function is obtained, and the vector feature is mapped using the mapping function to obtain the first control information, wherein the mapping function is determined during the training process of a prediction network, and the prediction network is used to implement the control method.

4. The method according to claim 1, characterized in that The first control information includes a first operation point, a first moving distance, and a first moving direction for the target object, and controlling the electronic device to perform an expansion operation on the target object based on the first control information includes: Controlling a mechanical arm of the electronic device to perform a grabbing operation on the first operating point; Controlling the robotic arm to move the first moving distance in the first moving direction; The robot arm is controlled to perform a release operation on the first operating point.

5. The method according to any one of claims 1 to 4, characterized in that: The control method is implemented by a trained prediction network, and the method further comprises: Obtain the initial prediction network, image samples of training objects, and sample labels; Using the initial prediction network to encode the image sample to obtain a sample encoding result; Performing an expansion prediction process based on the sample encoding result to obtain second control information; Based on the second control information and the sample label, the model parameters of the initial prediction network are adjusted to obtain the trained prediction network.

6. The method according to claim 5, characterized in that The initial prediction network includes a coding model, a first prediction model and a second prediction model. The using the initial prediction network to encode the image sample to obtain the sample encoding result includes: using the encoding model to encode the image sample to obtain the sample encoding result; The performing the expansion prediction process based on the sample encoding result to obtain the second control information includes: Using the first prediction model to perform motion vector prediction on the sample encoding result to obtain a second motion vector; The second prediction model is used to perform control information prediction on the second motion vector to obtain second control information.

7. The method according to claim 6, characterized in that The sample label includes a motion vector label and a control information label, and adjusting the model parameters of the initial prediction network based on the second control information and the sample label to obtain the trained prediction network includes: Determining a first loss value based on the second motion vector and the motion vector label, and adjusting a model parameter of the first prediction model based on the first loss value until a first training termination condition is reached, thereby obtaining a trained first prediction model; Determine a second loss value based on the second control information and the control information tag, and adjust a model parameter of the second prediction model based on the second loss value until a second training termination condition is reached, thereby obtaining a trained second prediction model; The trained prediction network is constructed using the encoding model, the trained first prediction model and the trained second prediction model.

8. The method according to claim 5, characterized in that The training object includes a second operating point and M marking points, where M is a positive integer, and the method further includes: Acquire a first starting position of the second operating point and second starting positions of the M marking points before a folding operation is performed on the training object based on the second operating point, and a second image of the training object, a first ending position of the second operating point, and second ending positions of the M marking points after the folding operation is performed; determining a third movement vector based on the first starting position and the first ending position; For each of the marking points, based on the second starting position of the marking point and the second ending position of the marking point, determining a fourth movement vector of the marking point; The second image is determined as the image sample, and the sample label is constructed based on the second operating point, the third motion vector and M fourth motion vectors.

9. The method according to claim 8, characterized in that The third movement vector includes a second movement distance and a second movement direction, the fourth movement vector includes a third movement distance and the second movement direction, and constructing the sample label based on the second operating point, the third movement vector and M fourth movement vectors includes: determining the opposite direction of the second moving direction as a third moving direction; constructing a fifth movement vector using the second movement distance and the third movement direction; constructing M sixth movement vectors using M third movement distances and the third movement directions; Determine the M sixth motion vectors as motion vector labels, and determine the second operation point and the fifth motion vector as control information labels; The motion vector label and the control information label are determined as the sample labels.

10. The method according to any one of claims 1 to 4, characterized in that: After controlling the electronic device to perform an expansion operation on the target object based on the first control information, the method further includes: acquiring a third image of the target object; When the third image represents that the target object is in a folded state, encoding the third image to obtain a second encoding result; Predicting the seventh motion vectors of the N marking points when performing an expansion operation on the target object based on the second encoding result, mapping the N seventh motion vectors to obtain third control information of the electronic device; Based on the third control information, the electronic device is controlled to perform an expansion operation on the target object until the target object reaches an expanded state.

11. A control device, characterized in that: The device comprises: A first acquisition module, used to acquire a first image of the target object in a folded state, wherein the target object includes N marking points, where N is a positive integer; A first encoding module, used to perform encoding processing on the first image to obtain a first encoding result; A first prediction module, configured to predict, based on the first encoding result, first motion vectors of the N marking points when performing an expansion operation on the target object, and perform mapping processing on the N first motion vectors to obtain first control information of the electronic device; The first control module is used to control the electronic device to perform an expansion operation on the target object based on the first control information.

12. A computer device, characterized in that: The computer device comprises: A memory for storing computer executable instructions; A processor, configured to implement the control method according to any one of claims 1 to 10 when executing the computer executable instructions stored in the memory.

13. A computer-readable storage medium storing computer-executable instructions, characterized in that: When the computer executable instructions are executed by a processor, the control method according to any one of claims 1 to 10 is implemented.

Images