Target control method and device based on multi-modal information, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence, can be applied to business scenes such as mechanical arm grabbing, financial science and technology and medical health, and discloses a target control method, device and equipment based on multi-modal information and a medium, and the method comprises the steps that a language instruction expressing an operation intention is obtained, and visual information of the environment where a target object is located is collected; semantic representation is generated through the language processing module, and visual features are generated through the visual processing module; inputting the visual features and the semantic representations into an action generation module to generate an action sequence; and a joint angle sequence and a joint angle value are determined based on the action sequence, and the mechanical arm is controlled to execute grabbing operation. According to the method, the accurate grabbing action sequence is generated by fusing language and visual information, the movement of the mechanical arm is controlled by combining the joint angle sequence and the accurate joint angle value, and smooth track movement and accurate grabbing are achieved. The adaptive capacity and the industrial-grade grabbing precision in the complex environment are improved, and the production efficiency and the grabbing success rate are effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a target manipulation method, device, equipment, and storage medium based on multimodal information. Background Art

[0002] Current robotic arm grasping technology faces many challenges, mainly including high-precision requirements, computing speed bottlenecks, poor environmental adaptability, and insufficient system generalization capabilities. In terms of precision, existing technologies cannot meet the requirements for grasping tiny and precise parts, especially in the industrial, medical, and financial fields. In terms of computing speed, existing systems usually take a long time, affecting the real-time response capabilities of scenarios such as high-speed production lines. In terms of environmental adaptability, external factors (such as changes in lighting and object posture) have a greater impact on the visual perception of the robotic arm, resulting in positioning errors, which in turn affect grasping accuracy. In terms of generalization ability, traditional systems cannot effectively grasp new objects that have not appeared in training, requiring a lot of retraining and debugging, which limits their use in complex application scenarios.

[0003] In the field of industrial robotics, traditional robotic arm technology cannot meet the needs of high-precision grasping, especially when handling tiny parts. Existing point cloud registration methods are unable to achieve submillimeter accuracy. For precise grasping in complex environments, traditional methods rely on CPU calculations, resulting in slow positioning speeds and unable to meet the responsiveness requirements of the robotic arm on high-speed production lines. In addition, changes in ambient lighting and object posture can lead to visual perception errors, significantly reducing the accuracy of the robotic arm when performing grasping tasks. These issues make it difficult for traditional technologies to cope with the diverse and complex grasping needs of industry, especially when efficient and precise operations are required.

[0004] In the healthcare sector, robotic gripping requires extremely high precision, especially during minimally invasive surgery, where even the slightest error can impact the outcome. Existing technologies are unable to maintain stable positioning accuracy in environments with fluctuating lighting and object posture, posing a significant challenge to the grasping of precision medical devices. Traditional grasping technologies exhibit poor adaptability to the diversity and irregularities of medical equipment, and are unable to quickly respond to new, unseen objects. Traditional methods also suffer from long computation times, making them particularly unsuitable for scenarios requiring high real-time performance, such as surgery.

[0005] In the financial sector, robotic arm grasping tasks involve high-precision equipment and hardware, such as bank teller machines and financial hardware. Existing technologies lack the precision to meet the requirements of financial hardware, especially when handling small, complex devices. Existing methods struggle to achieve the required accuracy. Furthermore, due to the diversity of financial equipment and environmental uncertainty, existing technologies have limited adaptability to these new objects, requiring extensive retraining and debugging. This significantly limits the application of existing technologies in the financial sector, making them ineffective for grasping tasks under diverse equipment and environmental conditions. Summary of the Invention

[0006] The main purpose of the present invention is to provide a target manipulation method, device, equipment and storage medium based on multimodal information, aiming to solve the technical problems that existing robotic arm grasping technology cannot simultaneously achieve end-to-end fusion of language command understanding, visual environment perception and action generation, resulting in low grasping accuracy, poor environmental adaptability and weak generalization ability.

[0007] To achieve the above objectives, the present invention provides a target manipulation method based on multimodal information, comprising:

[0008] Obtain language instructions that express operational intentions and collect visual information about the environment where the target object is located;

[0009] inputting the language instruction into a language processing module, and generating a semantic representation through the language processing module;

[0010] Inputting the visual information into a visual processing module, and generating visual features through the visual processing module;

[0011] Inputting the visual features and the semantic representation into an action generation module, and generating an action sequence through the action generation module;

[0012] determining a joint angle sequence and joint angle values ​​based on the action sequence;

[0013] The robotic arm is controlled based on the joint angle sequence and the joint angle values ​​to perform a grasping operation on the target object.

[0014] Furthermore, to achieve the above-mentioned purpose, the present invention provides a target manipulation device based on multimodal information, comprising:

[0015] The sensor module is used to obtain language instructions that express the operation intention and collect visual information about the environment where the target object is located;

[0016] a language processing module, configured to input the language instruction into the language processing module and generate a semantic representation through the language processing module;

[0017] a visual processing module, configured to input the visual information into the visual processing module and generate visual features through the visual processing module;

[0018] An action generation module, configured to input the visual features and the semantic representation into the action generation module, and generate an action sequence through the action generation module;

[0019] a kinematics calculation module, configured to determine a joint angle sequence and joint angle values ​​based on the motion sequence;

[0020] A control module is used to control the robotic arm based on the joint angle sequence and the joint angle value to perform a grasping operation on the target object.

[0021] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer device, which includes a memory, a processor, and a target control program based on multimodal information stored in the memory and executable on the processor. When the target control program based on multimodal information is executed by the processor, the steps of the target control method based on multimodal information as described above are implemented.

[0022] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, on which a target control program based on multimodal information is stored. When the target control program based on multimodal information is executed by a processor, the steps of the target control method based on multimodal information as described above are implemented.

[0023] Beneficial effects: The present invention relates to the field of artificial intelligence technology and can be applied to business scenarios such as robotic arm grasping, financial technology, and medical health. It discloses a target manipulation method, device, equipment, and medium based on multimodal information, including: obtaining language instructions that express the operation intention and collecting visual information of the environment where the target object is located; generating semantic representations through the language processing module, and generating visual features through the visual processing module; inputting the visual features and semantic representations into the action generation module to generate an action sequence; determining the joint angle sequence and joint angle values ​​based on the action sequence, and controlling the robotic arm to perform the grasping operation. The present invention generates an accurate grasping action sequence by fusing language and visual information, and controls the movement of the robotic arm in combination with the joint angle sequence and the precise joint angle value, thereby achieving smooth trajectory movement and precise grasping. It improves the adaptability and industrial-grade grasping accuracy in complex environments, and effectively improves production efficiency and grasping success rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0025] Figure 1 Schematic diagram of an application environment of a method for controlling an object based on multimodal information in an embodiment of the present invention;

[0026] Figure 2 1 is a flow chart of an embodiment of a method for controlling an object based on multimodal information according to the present invention;

[0027] Figure 3 Schematic diagram of functional modules of a preferred embodiment of a target control device based on multimodal information of the present invention;

[0028] Figure 4 A schematic diagram of the structure of a computer device according to an embodiment of the present invention;

[0029] Figure 5 FIG. 2 is another structural diagram of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0030] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0031] The target control method based on multimodal information provided by the embodiment of the present invention can be applied in the following situations: Figure 1 In an application environment, the user terminal communicates with the server terminal through a network. The server terminal can obtain language instructions that express the operation intention through the user terminal and collect visual information of the environment where the target object is located; generate semantic representation through the language processing module, and generate visual features through the visual processing module; input the visual features and semantic representation into the action generation module to generate an action sequence; determine the joint angle sequence and joint angle values ​​based on the action sequence, and control the robot arm to perform the grasping operation. The present invention generates an accurate grasping action sequence by fusing language and visual information, and controls the movement of the robot arm in combination with the joint angle sequence and the precise joint angle value to achieve smooth trajectory movement and precise grasping. It improves the adaptability and industrial-grade grasping accuracy in complex environments, and effectively improves production efficiency and grasping success rate. Among them, the user terminal can be but is not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The server terminal can be implemented with an independent server or a server cluster composed of multiple servers. The present invention is described in detail below through specific embodiments.

[0032] See also Figure 2 , Figure 2 This is a flowchart of an embodiment of a method for controlling an object based on multimodal information provided by the present invention. It should be noted that although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0033] like Figure 2 As shown, the target manipulation method based on multimodal information proposed in the present invention includes the following steps:

[0034] S10, obtaining a language instruction expressing the operation intention and collecting visual information of the environment where the target object is located;

[0035] In this embodiment, obtaining language instructions that express the operation intention and collecting visual information about the environment in which the target object is located is the basis for target manipulation. First, language instructions are an important medium for describing the user's intention, enabling the system to understand the manipulation task and the operation method of the target object. Language instructions are usually obtained through voice input devices or text input devices. Their content can involve actions such as moving, grabbing, and rotating. These instructions are parsed by the language processing module and converted into a command form that the machine can understand.

[0036] In actual implementation, the voice data collected by the voice input device is first converted into text by the voice recognition system, and then semantically analyzed by the natural language processing module. This module uses processes such as word segmentation and syntactic analysis to obtain a specific representation of the user's intent. For complex sentences or multiple instructions, semantic analysis is not just about converting language into text; it also requires abstracting key information such as the operation target, direction, and action method. For example, if the user's voice command is "Please pick up this cup and place it on the table," after analysis, the system can determine that the target object is "cup," the operation is "pick up," and the target location is "on the table." Through this process, semantic information can be converted into the actual content of the target operation.

[0037] At the same time, visual information of the environment in which the target object is located is a key data source to support the precise execution of operations. Through different sensor devices, such as optical cameras and depth sensors, the system can obtain rich environmental information. During the implementation process, visual information is collected by these devices, usually including two-dimensional images, depth data, and three-dimensional spatial coordinates of the target object. The two-dimensional image information obtained by the optical camera is combined with the three-dimensional point cloud data collected by the depth sensor to form comprehensive visual information of the target object. After passing through the image processing module, this information can provide data such as the shape, position, size, and posture of the object. The processing of visual information is not just a simple recognition of the image, but also includes complex processing of multiple steps such as image posture alignment, depth information denoising, and edge extraction of the target object to ensure that objects in the environment are accurately identified.

[0038] In different scenarios, the specific implementation can vary depending on the device configuration and actual needs. In some industrial applications, the system can be operated by a fixed desktop sensor, which directly provides the system with visual information of the target object and its surroundings. The system analyzes the image in real time and generates precise spatial coordinates based on the object's features in the image, allowing the control system to update operational data in real time.

[0039] For mobile environments or non-stationary targets, portable devices can be used for data collection. In this case, the data collected by the image and depth sensors is instantly transmitted to the processing unit for rapid analysis. By utilizing more efficient image processing algorithms and optimized depth sensing technology, the system can adapt to varying lighting conditions and object postures. As the environment changes, the system can dynamically adjust its operational strategy based on visual information and semantic instructions to ensure efficient and precise task completion in complex environments.

[0040] In the multimodal information processing process, deep learning algorithms can be combined with convolutional neural networks (CNNs) for image analysis to improve target recognition accuracy, and recurrent neural networks (RNNs) can be used to process time series data to improve the real-time and flexibility of task execution. In addition, sensor data fusion technology is used to effectively combine visual and language information, further enhancing the system's perception of the operating environment and ensuring the precise execution of various tasks.

[0041] Example: In a robotic arm grasping scenario, the system can interpret task requirements through user voice commands, such as "Please place the red cup on the table," and accurately identify the red cup's three-dimensional position based on environmental information captured by the camera. By combining visual information of the target object, the system generates a corresponding grasping path and controls the robotic arm to perform a precise grasping operation. During the grasping operation, the robotic arm adjusts its motion path in real time based on the target object's position and posture, as well as the target given by the voice command, ensuring it avoids any obstacles during movement and grasps the target object at a precise angle.

[0042] In healthcare, this same concept applies to the precision operations of medical robots. For example, during surgical assistance, doctors use voice commands to clearly define surgical objectives, while visual sensors provide real-time images and 3D data of the patient. By combining voice commands with visual information about the environment, medical robots can precisely control the tool's trajectory and accurately perform surgical operations.

[0043] In the financial sector, this technology can be used to automate item management in office environments. The system identifies target documents or items based on voice commands and uses visual perception to scan the office environment to accurately locate the items. This system can significantly improve item management efficiency in office environments, reduce operational errors, and minimize human intervention in processes such as item handling and document archiving, thereby increasing automation.

[0044] Through multimodal information fusion, this embodiment enables the system to fully understand the user's operational intent and accurately capture environmental data to form a complete operational execution plan. By combining voice commands with visual information, the system's adaptability and execution accuracy in complex environments are significantly improved. The system can adjust its strategies in real time in dynamically changing environments to ensure accurate grasping of target objects. At the same time, through the complementary nature of vision and language, errors during operation are greatly reduced, improving the success rate of grasping tasks.

[0045] S20, inputting the language instruction into a language processing module, and generating a semantic representation through the language processing module;

[0046] In this embodiment, the process of inputting language instructions into the language processing module first involves receiving natural language input from the user. These language instructions usually come from the user's spoken or written expressions, and they can be simple commands or complex multi-level requests. In order to convert natural language into commands that the computer can understand and execute, the input instructions need to be converted into a machine-processable text format through a speech recognition or text input processing module. The speech input is first processed by the speech recognition system to convert the speech signal into text. The text input is then directly transmitted to the subsequent processing module. Through this processing, the system can obtain semantically rich text data, which contains the specific content, intention and related parameters of the instruction.

[0047] The input language instructions then enter the language processing module. In the language processing module, the text will first pass through the word segmenter to segment the input long text into meaningful units, such as words, phrases, etc. After further grammatical analysis, these units are converted into internal representations that can be understood by the machine. A common technology used in this module is word embedding technology in natural language processing (NLP), which maps each word into a vector of fixed dimension. Through these vectors, the system can capture the similarity and correlation between words, providing important information for subsequent operations. For example, the instruction "bring the cup to the table" will be broken down by the system into several key components: "cup", "bring", and "table", and the action, target, and target location will be determined through semantic analysis.

[0048] The generated semantic representation not only includes the text's word meanings but also captures deeper meanings through contextual understanding. This contextual information enables the system to handle polysemous words, synonyms, and content related to other sentences. For example, in an instruction like "put on the table," the system needs to understand the meaning of "put on" based on the context and associate it with the target location, the table. Leveraging contextual features, the language processing module converts natural language into a structured representation suitable for subsequent task execution, including clear execution intent.

[0049] The implementation of the language processing module may vary in different application scenarios. In simple applications, the language processing module may rely solely on basic rule matching and lexical analysis to generate semantic representations. This approach is suitable for simple commands with fixed grammatical patterns, such as basic action commands like "pick up" and "put down."

[0050] However, in complex scenarios, especially when dealing with open-ended conversations or multi-round instructions, the system may need to use more sophisticated natural language understanding models. For example, deep learning models such as recurrent neural networks (RNNs) or long short-term memory networks (LSTMs) can handle the semantic understanding of long texts and generate more accurate semantic representations based on the context. During this process, the model dynamically adjusts its understanding, incorporating previous conversation content to better parse complex commands.

[0051] Further technical implementation could involve the introduction of reinforcement learning or self-supervised learning techniques, enabling the system to gradually optimize its semantic parsing capabilities during use. For unclear or ambiguous instructions, the system can self-adjust based on past interaction experience to improve the accuracy and efficiency of parsing complex instructions.

[0052] In this embodiment, by inputting language instructions into the language processing module and generating semantic representations, the system can more accurately understand the user's operational intentions and perform subsequent actions based on these intentions. It can flexibly respond to complex and dynamic user needs. The generation of semantic representations not only provides clear execution instructions, but also provides a basis for subsequent task generation, decision-making, and optimization. Through semantic understanding, the system can extract accurate operational instructions from ambiguous or incomplete information, significantly improving the system's responsiveness and adaptability.

[0053] S30, inputting the visual information into a visual processing module, and generating visual features through the visual processing module;

[0054] In this embodiment, the process of inputting visual information into the visual processing module begins with image acquisition of the environment, which is usually completed by an optical camera mounted on a robotic arm or other device. These cameras can collect color images, depth information or other relevant visual data of the environment in which the target object is located. The collected visual data may include the shape, color, texture and spatial layout of the object. Usually, visual information is not just a two-dimensional image, but also includes additional data that may be obtained through other sensing devices (such as depth sensors), such as depth information and three-dimensional morphological data of the object.

[0055] This input visual information is then passed to the visual processing module for processing. The main task of the visual processing module is to analyze the image data and extract useful visual features. Visual features include, but are not limited to, an object's geometric shape, surface texture, color distribution, and spatial location. These features are typically derived through a series of image processing algorithms, such as edge detection, feature point extraction, depth information processing, and texture mapping.

[0056] In the visual processing module, image preprocessing is first performed. This may include operations such as denoising, illumination correction, and image enhancement to ensure that the image information acquired in complex environments (such as under varying lighting conditions) accurately reflects the characteristics of the target object. Next, using advanced image recognition algorithms such as deep convolutional neural networks (CNNs), the system can identify the basic features and important feature points of the object from the original image. These features may include information such as the object's shape, size, color, and texture, and can separate the target object from a complex background.

[0057] For complex environments, the visual processing module may also involve 3D image reconstruction. This process typically generates a 3D model of the target object by fusing image information or depth data from different angles. In some applications, the system may also use a feature pyramid network (FPN) to extract multi-scale texture features, making the visual information more detailed and comprehensive. Ultimately, the processed visual features are converted into a digital data structure for use by the subsequent action generation module or decision module.

[0058] The implementation of the visual processing module can vary depending on the application scenario. In some basic applications, the system can use traditional computer vision methods, such as edge detection and corner extraction, to extract visual features. This approach is simple and consumes less computational resources, making it suitable for relatively simple tasks, such as identifying the position of objects in a static environment.

[0059] For more complex applications, such as grasping objects in dynamic environments or against changing backgrounds, the visual processing module typically employs deep learning techniques, particularly convolutional neural networks (CNNs) and regional convolutional neural networks (R-CNNs). These networks can effectively identify and locate target objects and extract higher-level, more refined visual features. In this scenario, the visual processing module not only relies on data from a single image but may also utilize data from various sensors (such as lidar and depth sensors), achieving higher-dimensional information fusion.

[0060] In certain scenarios, such as healthcare, the visual processing module may also require detailed analysis of the target object's morphology and structure, especially when grasping delicate objects. By combining deep learning with traditional image processing methods, the system can automatically adapt to different environments and object types, thereby improving overall grasping accuracy and efficiency.

[0061] Example: In a robotic arm grasping application scenario, assuming the target object is a mobile phone, the system first uses a visual sensor (such as an RGB camera) to obtain image data from the environment. The mobile phone in the image may be affected by changes in lighting. The system uses the visual processing module to denoise and enhance the contrast of the image. It then uses a convolutional neural network (CNN) to extract feature information such as the phone's edges, corners, and texture. The processed visual features include data such as the phone's shape, size, edges, and surface texture. These visual features are passed to the action generation module for subsequent grasping strategy generation. Through the accurately extracted visual features, the system can determine the exact position and posture of the mobile phone in the environment, thereby providing the robotic arm with accurate target position information, ensuring obstacle avoidance during movement and ultimately accurate grasping.

[0062] In the healthcare field, vision processing modules can also be used to grasp medical instruments. For example, surgical robots need to precisely grasp and position surgical tools. The system uses a camera to capture images of the tools on the operating table. The vision processing module then extracts the tool's geometric and texture features from the images. These features are transmitted to the control system, which calculates the tool's specific position, ensuring the robot can accurately grasp the surgical tool and perform high-precision surgical operations.

[0063] In the financial sector, visual processing modules can be applied to automated securities trading systems to identify and extract key information from charts. In this scenario, chart recognition and analysis are crucial for making investment decisions. The system uses the visual processing module to extract visual features such as trend lines and key data points from charts. Combined with trading instructions, it generates action sequences to execute stock buy and sell operations. This process ensures rapid and accurate financial decision-making and effectively improves market responsiveness.

[0064] In this embodiment, by inputting visual information into the visual processing module and generating visual features, the system can extract representative and accurate object information from the original image or sensor data. It can accurately identify target objects in complex environments and can cope with changing environmental factors, such as changes in lighting and changes in object posture. The generated visual features provide precise input for subsequent action generation and control, ensuring that the robotic arm can perform tasks efficiently and accurately. Through highly integrated visual feature processing, the system can handle a variety of tasks in different scenarios, and has strong environmental adaptability, which greatly improves the accuracy and response speed of the system.

[0065] S40, inputting the visual features and the semantic representation into an action generation module, and generating an action sequence through the action generation module;

[0066] In this embodiment, visual features and semantic representations are input into the action generation module to generate an action sequence. Visual features are information such as object shape, color, and texture extracted from image data by the visual processing module, and contain information about the spatial position of the target object in the environment. The semantic representation is generated by the language processing module and reflects the intention of the operation instruction, such as the type of target object, the target operation, and the related action requirements. The action generation module fuses these two types of information, combining the spatial features of the visual information and the operational intention in the semantic representation to generate a continuous action sequence.

[0067] The core task of the action generation module is to convert the input visual features and semantic representations into executable robotic arm operation commands. This module first fuses the visual features and semantic representations. This fusion typically combines the spatial positioning information in the visual features with the operational intent in the semantic representation through a specific algorithmic model, forming a high-level feature representation that incorporates both visual and semantic information. This fusion process aims to unify environmental information and operational requirements, ensuring that the generated action sequence accurately reflects the spatial requirements and task intent of the operation.

[0068] During the process of generating an action sequence, the action generation module plans a series of motion instructions for the robotic arm based on the position and posture of the target object, as well as the type of grasping task. The action sequence includes not only the motion trajectory of the robotic arm, but also the operational control of the gripper, such as the closing force. This sequence determines the target angle of each joint through inverse kinematics calculations, ensuring that the robotic arm moves smoothly from the current state to the target position and accurately performs the grasping operation. The action generation module also needs to optimize the exact position of the grasping point to ensure that the target object is successfully grasped by the robotic arm. This optimization process usually needs to consider multiple factors such as the object posture, grasping position, and the current state of the robotic arm to generate an action sequence that adapts to complex environments.

[0069] The implementation of the motion generation module can be adjusted based on actual needs. For example, in a standard industrial environment, the motion generation module may need to respond quickly and generate precise motion sequences to ensure the robot arm can successfully perform tasks in complex environments. In a dynamically changing environment, however, the motion generation module may need to incorporate more adaptive adjustment mechanisms, such as adjusting the grasping path and control strategy based on real-time visual feedback, to improve the system's robustness and adaptability.

[0070] During the specific implementation process, first, the visual processing module extracts key features from the image through a deep learning model and converts them into visual feature vectors. The semantic representation generation module uses natural language processing technology to parse the input language instructions and converts them into semantic information with clear operation directions. After the two are combined, the action generation module generates executable action instructions by fusing this information. In industrial applications, the performance of the system can be improved through optimization algorithms, such as using a feature fusion method based on a graph neural network, or optimizing the output accuracy of the action generation model through a multi-layer perceptron (MLP). For specific application scenarios, the generated action sequence can also be adjusted according to the load of the robotic arm and the characteristics of the grasped object to ensure the stability and safety of the grasping action.

[0071] Example: In the healthcare field, in an automated drug sorting system, visual information of the drugs is first collected through cameras and depth sensors and converted into visual features. The user tells the system the type of item to be grabbed through voice commands (such as "grab the bottled medicine on the table"). The language processing module parses this command and generates a corresponding semantic representation. The visual features and semantic representation are input into the action generation module to generate a continuous action sequence to control the robotic arm to grab the target item. With this method, the system can perform tasks efficiently and accurately in a dynamically changing medical environment, avoiding the grasping errors caused by environmental interference in traditional methods.

[0072] In the financial sector, automated document processing systems need to identify and grasp various objects. In this system, visual information comes from a camera scanning documents, while voice commands, such as instructions from bank staff, may be instructions to grab certain documents or perform certain operations. By combining the semantic and visual features in the voice commands, an action generation module generates an action sequence to instruct the robotic arm to grab the target document and perform the corresponding operation.

[0073] This embodiment integrates visual features and semantic representations into an action generation module, which then generates action sequences. This allows the system to accurately perform grasping tasks in complex environments. This method combines visual and linguistic information, enabling the robotic arm to determine an object's position and form based not only on image features but also on the explicit operational intent of instructions, significantly improving the system's adaptability, accuracy, and real-time performance.

[0074] S50, determining a joint angle sequence and joint angle values ​​based on the action sequence;

[0075] In this embodiment, determining the joint angle sequence and joint angle values ​​based on the motion sequence is a key step in the robotic arm control process. The motion sequence is a continuous series of motion instructions generated from visual features and semantic representations, instructing the robotic arm on how to perform a grasping operation. The joint angle sequence and joint angle values ​​are calculated based on these motion instructions and the robotic arm's kinematic model to generate specific joint motion instructions.

[0076] First, each action in an action sequence represents a movement or operation of the robot arm, typically involving multiple waypoints. These waypoints contain the target position or angle of each joint of the robot arm at different moments in time. The joint angle sequence parses the waypoint information in the action sequence to calculate the corresponding joint angle for each waypoint. For each waypoint, an inverse kinematics algorithm is used to calculate a set of joint angles. These angles are key to controlling the robot arm's movement along a predetermined trajectory.

[0077] Inverse kinematics algorithms are used to calculate the angles of each joint in a robot or robotic arm to achieve the desired position and orientation of the end effector (such as a gripper or tool). Unlike forward kinematics, which calculates the end effector's position given known joint angles and positions, inverse kinematics calculates the required joint angles given both the end effector's position and orientation.

[0078] The core problem of inverse kinematics is how to infer the angles of each joint based on a given target position and posture. Each joint in a robotic arm has a specific range of motion. Therefore, when performing inverse kinematics calculations, reasonable joint angles must be determined based on the robotic arm's kinematic model, joint constraints, and the requirements of the end effector. This algorithm typically includes mathematical solution steps. Especially for robotic arms with multiple degrees of freedom, inverse kinematics may have multiple solutions or no solution. For example, certain target positions may not be reachable using certain joint angle combinations, or there may be multiple different angle combinations that can achieve the same target position.

[0079] During the calculation process, inverse kinematics often relies on geometric methods or numerical methods. Geometric methods use triangles and vector relationships in geometry to solve joint angles through step-by-step iterations; while numerical methods minimize the error between the actual end position and the target end position by continuously adjusting the joint angles until the accuracy requirements are met.

[0080] Since the calculations involved in inverse kinematics are usually complex, especially when the robot arm has a high degree of freedom, the solution may take a long time. Therefore, in practical applications, it is usually necessary to optimize the algorithm and combine it with technologies such as motion planning, collision detection, and path smoothing to enable the robot arm to complete complex operating tasks quickly and accurately. For example, in the grasping operation of an industrial robot, the inverse kinematics algorithm can calculate the angle that each joint should reach based on the spatial position and posture of the grasping point, ensuring that the robot arm can accurately position the end effector to the correct grasping position when performing the grasping task, thereby improving the success rate and accuracy of the grasping.

[0081] There are many ways to implement the inverse kinematics algorithm, including:

[0082] Analytical methods (analytical solutions): Analytical methods use geometric relationships to directly derive the solution for joint angles from known target positions and postures. This approach is suitable for simpler robotic arm structures, such as two-dimensional or three-dimensional robotic arms with few degrees of freedom. Analytical methods typically use trigonometric functions or algebraic equations to solve for the joint angles. For example, the inverse kinematics of a two-dimensional robotic arm can be directly solved using trigonometric methods.

[0083] Numerical method (numerical solution): The numerical method is to gradually approximate the solution of the joint angle through iteration. The most common numerical methods are the Newton-Raphson method, the gradient descent method, and the Levenberg-Marquardt algorithm. These methods iteratively adjust the joint angle based on the error between the position of the target end effector and the current configuration until the error is small enough. The Newton-Raphson method calculates the gradient of the error in the solution space and uses this gradient information to adjust the joint angle. The gradient descent method also calculates the gradient of the error and adjusts it in the direction of minimizing the error.

[0084] Iterative method (such as Jacobian inverse method): Iterative method is a way to approximate the solution through repeated updates. The Jacobian matrix is ​​a matrix that describes the relationship between the changes in each joint of the robot and the changes in the end effector. By calculating the inverse of the Jacobian matrix, the position change of the end effector can be converted into changes in the joint angle. When using the Jacobian inverse method, first calculate the error between the current end effector position and the target position, and then adjust the joint angle according to the inverse matrix of the Jacobian matrix, and iterate until the error is small enough. This method is generally suitable for multi-degree-of-freedom robotic arms, especially when an analytical solution is not available.

[0085] Optimization-based inverse kinematics: This approach solves the inverse kinematics problem by converting it into an optimization problem. Specifically, the goal is to minimize the error between the end-effector position and the target position. An optimization algorithm (such as genetic algorithms or simulated annealing) is used to continuously adjust the joint angles until the target position is sufficiently accurate.

[0086] Next, joint angles provide the final, precise control of the grasping action. These values ​​are typically associated with the last action or target position in a sequence and represent the specific angles the robot arm needs to reach to grasp the target object. These angles are generated based on the location of the grasp point, the current state of the robot arm, and the position and posture of the target object. Joint angles are typically adjusted upon reaching the target position to ensure the robot arm is accurately positioned to complete the grasping operation.

[0087] During implementation, inverse kinematics calculations determine the target angles for each joint based on the robot's link parameters, joint constraints, and target pose information. From this foundation, the joint angle sequence is used to plan the robot's trajectory, while the joint angle values ​​ensure precise positioning during grasping. To account for potential environmental interference or changes in the object's pose, multiple optimization calculations may be required to ensure that each joint angle adjustment meets actual operational requirements.

[0088] In the implementation process, pathpoint information is first extracted from the action sequence. Pathpoints are generated using visual information and semantic instructions, and contain the target object's position and the desired pose of the robotic arm. Based on the position information of each pathpoint, the inverse kinematics module calculates the inverse angle relationship of the robotic arm's joints to determine the corresponding joint angle sequence for each pathpoint. For each joint, the inverse kinematics algorithm calculates an angle value that enables the robotic arm to smoothly transition from its current position to the target position. Furthermore, for precise positioning of the grasping point, the robotic arm's end effector requires precise joint angle values ​​based on the target position specified in the action sequence. This angle value is typically optimized to ensure pose accuracy. Especially when performing gripping or grasping actions, the robotic arm must adjust joint angles within a narrow tolerance to ensure stable grasping of the target object. In industrial applications, implementations may consider using various optimization techniques, such as those based on genetic algorithms or particle swarm optimization, to further refine joint angle adjustment and ensure the robotic arm's adaptability and reliability in dynamic environments. This allows the robotic arm to adjust joint angles in real time based on environmental changes to adapt to the grasping requirements of different objects.

[0089] For example, in a robotic grasping application, suppose a bottle on a table needs to be grasped. First, the robotic arm uses a vision system to identify the bottle's location and generates an action sequence that instructs the arm to move from its current position to the target location. During this process, the joint angle sequence guides the arm to smoothly follow the predetermined path, avoiding potential obstacles. Finally, the arm precisely adjusts the joint angles to ensure the gripper maintains a stable grasp on the bottle. This series of precise control steps enables the robotic arm to quickly and accurately complete the grasping task, avoiding failure or damage due to operational errors.

[0090] In healthcare, similar technologies can be applied to automated drug sorting systems. Vision systems identify the location and type of medications, generating action sequences that robotic arms use to precisely grasp and place each vial, ensuring each bottle is accurately picked up and placed in its designated location. In finance, automated office systems use robotic arms to grasp and organize documents. Vision systems work in conjunction with voice commands to ensure documents are grasped and processed in the correct order.

[0091] By determining the joint angle sequence and values ​​based on the motion sequence, this system achieves more precise and smoother robotic arm motion, ensuring a smooth transition from the starting position to the target position. It also achieves submillimeter precision control when grasping the target object. This method effectively addresses the accuracy and real-time performance issues inherent in traditional robotic arm control, improving the robotic arm's adaptability in dynamic environments, particularly in complex industrial settings, enabling efficient and stable grasping tasks.

[0092] S60: Control the robotic arm based on the joint angle sequence and the joint angle values ​​to perform a grasping operation on the target object.

[0093] In this embodiment, when performing the target object grasping operation, the joint angle sequence and the joint angle value serve as two important parameters for controlling the movement of the robotic arm, which are responsible for the path movement and the precise positioning of the end point, respectively. The joint angle sequence is usually a set of path points generated by the action generation module according to the target position and the starting position of the robotic arm. The robotic arm moves along the path sequence to complete a smooth transition from the current position to the target object. The joint angle sequence in this process only needs to ensure the continuity and smoothness of the motion path, and ensure that the robotic arm can avoid obstacles in the environment. Therefore, the generation of the joint angle sequence emphasizes the smoothness of the trajectory and the rationality of the path, without excessive attention to accuracy.

[0094] However, when the robotic arm reaches a position close to the target object, the role of the joint angle value becomes crucial. The joint angle value is used to fine-tune the position of the end effector of the robotic arm to ensure that the gripper can align the target object at the correct angle for grasping. At this time, the joint angle value is calculated through inverse kinematics, and high-precision calculations are usually required to ensure that the end effector can accurately dock the target object to avoid grasping failure or deviation. Due to the very high requirements for grasping accuracy, the calculation of the joint angle value generally requires fine-tuning using optimization algorithms (such as gradient descent method, Levenberg-Marquardt optimization, etc.) to ensure that the accuracy of the final calculation result meets industrial-grade grasping standards.

[0095] By precisely controlling the joint angles, the gripper can close stably and accurately as the target object approaches. The key to this operation lies in the accuracy of the joint angles, which not only influences the gripper's ability to grasp the object but also determines the stability of the grip. The robot arm must smoothly move along the path by controlling the joint angle sequence before reaching the target object. At the endpoint, fine-tuning the joint angles ensures the gripper closes in the precise position.

[0096] Joint angle sequences and joint angle values ​​complement each other and work together to complete grasping operations. The former is responsible for the robot's path planning and motion control, while the latter ensures that the end effector makes fine adjustments as it approaches the target, achieving precise grasping. This combination effectively improves the robot's adaptability in complex environments, enabling it to accurately grasp target objects in dynamic environments. In practical applications, the robot arm can use joint angle sequences to complete trajectory control and combine joint angle values ​​for precise target alignment and grasping operations, ensuring stable motion and high-precision grasping.

[0097] In specific implementation, first of all, in an industrial production environment, the joint angle sequence of the robotic arm is usually generated by an action generation module, which calculates and generates a continuous set of path points based on the pose information of the target object (including position coordinates and attitude angles) and the current state of the robotic arm. This set of path points forms a joint angle sequence. In order to adapt to different operating environments and work objectives, the density of the path points can be adjusted when generating the joint angle sequence. The denser the path points, the smoother the motion trajectory of the robotic arm, but the amount of calculation will also increase accordingly. Therefore, in industrial applications, the appropriate path point density can be selected according to the requirements of the task to balance the calculation speed and accuracy requirements. The joint angle sequence is used to control the smooth transition of the robotic arm from the current position to the target position, ensuring that the robotic arm can avoid obstacles in the working environment and reach the target position smoothly.

[0098] Based on the joint angle sequence, the joint angle value plays a vital role when the robot arm approaches the target object. The joint angle value is used to fine-tune the position of the end effector of the robot arm to ensure that the gripper docks with the target object at the appropriate angle for grasping. At this time, the calculation of the joint angle value usually adopts the inverse kinematics algorithm, and because the accuracy required for grasping is extremely high, it is usually combined with an optimization algorithm (such as gradient descent or Levenberg-Marquardt optimization) for fine adjustment. The optimization algorithm iteratively calculates according to the current position of the robot arm and the final position of the target object until the accuracy meets the requirements, ensuring that the end effector can close the gripper at the most precise position to complete the grasping task.

[0099] In terms of the selection of application scenarios, the calculation method of joint angle sequences and joint angle values ​​will be customized according to the specific requirements of the medical and health fields and the financial fields. In the medical and health field, the robotic arm may need to perform subtle operations in more complex environments, such as grasping surgical tools, picking up objects, and other tasks, which requires higher grasping accuracy and operational stability. In this case, the path points generated by the joint angle sequence may need to be more dense to ensure the smoothness of the path. At the same time, the optimization calculation of the joint angle value will use a more sophisticated algorithm to ensure high precision of the action and avoid any minor errors during the operation.

[0100] In the financial sector, particularly in logistics and warehousing, robotic arm grasping tasks may involve fast and efficient object handling. For example, in large warehouses, robotic arms must quickly grasp and move goods. In this case, joint angle sequence generation can prioritize speed, with a low density of path points to improve efficiency. Joint angle calculations are primarily used to ensure the end effector accurately aligns with the target object for grasping, avoiding errors or damage caused by rapid grasping.

[0101] The algorithms and technical parameters within the implementation can be flexibly adjusted to meet the needs of different applications. For example, optimizing the inverse kinematics solution process can speed up or improve calculation accuracy, while adjusting the complexity of the path planning algorithm can optimize the efficiency of the robot's motion. For different robot arm types, the appropriate calculation method can be selected based on the arm's load capacity, joint degrees of freedom, and flexibility, ensuring the optimal grasping operation for a specific task.

[0102] Finally, in real-world industrial environments, feedback mechanisms during execution are equally important. By embedding a real-time feedback module in the joint control system, the robot arm can automatically adjust its path and motion based on the real-time results of the grasping operation, ensuring successful task completion. In the event of a failed grasp, re-grasping can be achieved by replanning the joint angle sequence and angle values, thus ensuring the reliability and efficiency of the entire process.

[0103] This embodiment can achieve efficient grasping of the robot arm in a complex environment through precise control based on the joint angle sequence and joint angle value. The joint angle sequence ensures that the robot arm moves along a smooth path, avoids obstacles and reaches the target position smoothly, while the joint angle value ensures that the gripper can complete the grasping task with sub-millimeter precision by fine-tuning the position of the end effector. This precise path control and grasping accuracy not only improves the operating efficiency of the robot arm, but also enhances its application capabilities in scenarios with high precision requirements. In many fields such as industrial production, medical health, and financial logistics, the work efficiency of the robot arm and the accuracy of task execution can be significantly improved, thereby effectively meeting the requirements of the robot arm's grasping accuracy, speed and stability in different environments.

[0104] The present invention relates to the field of artificial intelligence technology and can be applied to business scenarios such as robotic arm grasping, financial technology, and medical health. It discloses a target manipulation method, device, equipment, and medium based on multimodal information, including: obtaining language instructions that express the operation intention and collecting visual information of the environment where the target object is located; generating semantic representations through a language processing module, and generating visual features through a visual processing module; inputting the visual features and semantic representations into an action generation module to generate an action sequence; determining the joint angle sequence and joint angle values ​​based on the action sequence, and controlling the robotic arm to perform a grasping operation. The present invention generates an accurate grasping action sequence by fusing language and visual information, and controls the movement of the robotic arm in combination with the joint angle sequence and the precise joint angle values, thereby achieving smooth trajectory movement and precise grasping. It improves the adaptability and industrial-grade grasping accuracy in complex environments, and effectively improves production efficiency and grasping success rate.

[0105] In one embodiment, the above step S10 includes:

[0106] S101, receiving a voice input for expressing an operation intention, and converting the voice input into a natural language instruction in text form;

[0107] S102, collecting a color image of the environment where the target object is located by using an optical camera, and collecting depth information of the environment where the target object is located by using a depth sensor;

[0108] S103, converting the depth information into three-dimensional point cloud spatial coordinate data, and performing outlier filtering processing on the three-dimensional point cloud spatial coordinate data;

[0109] S104: performing spatial pose alignment processing on the three-dimensional point cloud spatial coordinate data after outlier filtering processing and the color image to generate the visual information.

[0110] In this embodiment, obtaining language instructions that express the operating intention and collecting visual information of the environment where the target object is located is to achieve precise target manipulation. First, voice input is received and converted into natural language instructions in text form. Through the speech recognition module, the voice input is converted into text information that can be semantically analyzed. This process usually involves the cooperation of acoustic models and language models, wherein the acoustic model is used to convert sound signals into phonemes or words, while the language model is responsible for processing the grammatical structure and contextual information in the text, and ultimately forming natural language instructions that can express the user's intention. The accuracy of speech recognition is crucial to the successful execution of the task.

[0111] Secondly, an optical camera captures a color image of the target object's environment, and a depth sensor acquires depth information about that environment. This color image provides the robotic arm with visual features of the target object, capturing information such as its shape and texture, and assisting with subsequent object recognition and positioning. Depth sensors typically use laser or infrared technology to measure the distance to an object and generate a depth map. This depth data is crucial for the robotic arm to determine the object's three-dimensional spatial position and effectively handle objects at varying distances.

[0112] After processing, depth information is converted into three-dimensional point cloud spatial coordinate data. Point cloud data consists of multiple spatial points, each representing a location in space, typically including its X, Y, and Z coordinates. A key step in the point cloud conversion process is outlier filtering. This is used to remove abnormal points caused by sensor noise, reflections, or poor sensor data. This process can use statistical methods such as standard deviation filtering and density clustering to identify and remove points that do not conform to the spatial structure, thereby improving the quality of the point cloud data and the accuracy of subsequent processing.

[0113] Finally, the spatial coordinate data of the 3D point cloud, after outlier filtering, is aligned with the color image. Spatial pose alignment involves registering point cloud data and image information in 3D space, so that the visual and depth information of the same object are represented in the same coordinate system. This process typically uses feature matching algorithms, such as SIFT (Scale-Invariant Feature Transform) or ORB (Oriented FAST and Rotated BRIEF), to ensure that the image's texture information is aligned with the spatial information of the 3D point cloud, thereby improving the accuracy of object recognition and positioning.

[0114] This embodiment, by combining voice commands and visual information, can achieve a more natural way of inputting operational intentions and more accurate environmental perception. The processing of language commands ensures that the user's intentions can be correctly understood by the system, while the collection and processing of visual information provides accurate three-dimensional positioning of objects. Through the collection of depth information and the processing of point cloud data, the system can accurately identify target objects in dynamic or complex environments and effectively track and locate them. The application of outlier filtering improves the quality of the data, allowing grabbing operations to be performed in a high-precision environment.

[0115] In one embodiment, the above step S20 includes:

[0116] S201, performing word segmentation processing on the language instruction by a language processing module to generate a plurality of word units;

[0117] S202, converting each word unit into a word embedding vector through the language processing module to obtain a word embedding vector sequence;

[0118] S203, processing the word embedding vector sequence through a recurrent neural network in the language processing module to generate context features;

[0119] S204: Process the context features through the language model in the language processing module to generate a semantic representation.

[0120] In this embodiment, the process of inputting language instructions into a language processing module and generating semantic representations through the module is intended to convert natural language instructions into task instructions that can be understood and executed by a computer. First, the language instructions need to be processed by word segmentation. Word segmentation is the process of dividing a continuous text into smaller units with independent meanings, namely word units. This processing is a basic step in text understanding and can be performed through a dictionary-based rule method or statistical model. The word units after word segmentation will be further processed to ensure that the machine can accurately understand the meaning of each word.

[0121] Next, the segmented word units are converted into word embedding vectors through the language processing module. Word embedding is a method for converting words into vector representations, so that semantically similar words have similar representations in the vector space. Common word embedding methods include Word2Vec and GloVe. By training on large-scale text data, they can generate high-dimensional vectors for each word, and these vectors can capture the semantic relationships between words. These word embedding vectors constitute a word embedding vector sequence and serve as input for subsequent processing.

[0122] The recurrent neural network (RNN) in the language processing module then processes the sequence of word embedding vectors. RNNs are a type of neural network suitable for processing sequential data, capable of learning contextual information based on the preceding and following relationships in the input sequence. In language processing tasks, RNNs are effective in capturing the grammatical structure and semantic context of sentences, such as the relationships and dependencies between words. Through RNN processing, the system generates contextual features that contain contextual information, providing a deep understanding of the semantics of the language instructions.

[0123] Finally, the language model further processes the generated contextual features. A language model is trained using large amounts of text data and is capable of understanding the overall structure and common patterns of language. During this step, the language model analyzes the contextual features and generates a semantic representation. A semantic representation is an abstract, structured representation that captures the true meaning of language instructions and provides a basis for subsequent task generation or control. This semantic representation not only contains the semantic information of each word but also understands the relationships between words, helping the system perform more complex tasks.

[0124] Through this series of processes, the system in this embodiment can effectively convert natural language instructions into structured semantic representations, enabling the robotic arm to understand and execute the user's operational intentions. The combination of word segmentation processing and word embedding vectors provides the basis for semantic analysis, while RNN and language models help the system capture context and deep semantic structure. Ultimately, the generated semantic representation enables the system to accurately identify operational targets and perform corresponding actions in complex scenarios, significantly improving the accuracy of language instruction comprehension and the flexibility of mechanical operations.

[0125] In one embodiment, the above step S30 includes:

[0126] S301, separating a mask region of a target object from the visual information by a visual processing module;

[0127] S302, processing the mask area through a convolutional neural network in the visual processing module to obtain geometric shape features;

[0128] S303, processing the mask area through a feature pyramid network in the visual processing module to obtain multi-scale texture features;

[0129] S304: Fusing the geometric shape features and the multi-scale texture features through the visual processing module, and generating visual features based on the fused features.

[0130] In this embodiment, the process of inputting visual information into the visual processing module is to convert the image or video stream information obtained from the environment into useful features that can be used to control equipment such as robotic arms. In this step, it is first necessary to separate the mask area of ​​the target object from the visual information through the visual processing module. The mask area refers to the part of the image related to the target object. This process usually relies on target detection technology, such as object detection methods based on deep learning (such as YOLO, Faster R-CNN, etc.). Through these algorithms, the boundaries of objects in the image can be identified and framed, and then the corresponding mask area can be obtained, which is convenient for subsequent feature extraction.

[0131] Next, a convolutional neural network (CNN) processes the extracted masked area to extract geometric features. CNN is a deep learning model that excels at processing image data and can automatically extract spatial features of images through convolutional layers. Geometric features include information such as an object's edges, corners, and shape outlines. These features are crucial for identifying an object's position and posture. For example, when processing an image of a cube, a convolutional neural network can automatically learn the object's edges, corners, and faces and generate a corresponding geometric description.

[0132] In addition, the Feature Pyramid Network (FPN) further processes the masked region and extracts multi-scale texture features. FPN is a network architecture for image processing specifically designed for multi-scale feature extraction. Through multi-level convolution operations, it captures feature information at different scales, effectively processing objects of varying sizes. In this process, FPN extracts texture information at different scales through bottom-up multi-level convolutions. This information reflects surface details such as texture and smoothness.

[0133] Finally, the visual processing module fuses the geometric shape features with the multi-scale texture features and generates the final visual features based on these fused features. This fusion process can be performed using methods such as weighted averaging, concatenation, or deep neural networks, combining information from different feature levels to obtain a comprehensive object representation. The resulting visual features contain not only the object's shape information but also its texture and level of detail information, providing more accurate input for subsequent object localization and grasping operations.

[0134] Example: Consider an image containing multiple objects (for example, a large box, a small bottle, and a few scattered small items). Each object varies greatly in size, texture, and shape. The task of an object detection algorithm is to identify and locate these objects, and it must be able to handle objects of varying sizes.

[0135] Input image: The size of the input image is 1024x1024 pixels and contains multiple objects.

[0136] Convolution operation: The image is first processed through a convolutional neural network (CNN). The first stage of a CNN usually downsamples the image (for example, through max pooling) to extract low-level features (such as edges, textures, and colors).

[0137] Feature Pyramid Construction: The core idea of ​​FPN is to capture image features at different scales through multi-layered feature maps. Traditional CNNs capture detailed features (such as texture) at shallow layers, but these features are low-resolution and suitable for recognizing small objects. Deeper features typically capture global information but are less detailed, making them suitable for recognizing large objects.

[0138] Shallow features: Details are captured by feature maps of low-level convolutional layers (such as 128x128 or 64x64).

[0139] Deep features: Global information is captured through deeper convolutional layers (such as 32x32 or 16x16).

[0140] Feature Pyramid Merging: FPN uses a bottom-up network structure to combine feature maps from each layer to form a multi-scale feature pyramid. Shallow feature maps typically have higher spatial resolution but lower semantic information, while deep feature maps contain more semantic information but lower resolution. FPN uses a top-down approach to fuse low-resolution deep feature maps with high-resolution shallow feature maps.

[0141] Multi-scale prediction: In FPN, feature maps are processed layer by layer through convolution to produce multi-scale outputs. Each scale of the output feature map has its own applicable object size:

[0142] Small objects: Shallow feature maps (higher resolution) are best suited for identifying small objects such as bottles, keys, etc.

[0143] Large objects: Deep feature maps (lower resolution) are used to identify large objects such as boxes, sofas, etc.

[0144] Output: The FPN generates a corresponding prediction box for each scale feature map, and the Region Proposal Network (RPN) performs object classification and localization. Features at multiple scales can improve the detection accuracy of small and large objects.

[0145] This embodiment effectively helps the system identify target objects and determine their position and posture by converting visual information into rich visual features. By separating masked regions and extracting geometric and texture features, the system can extract more detailed information from images. The fused visual features enhance the system's ability to perceive objects, providing higher recognition accuracy, particularly in complex environments. These enhanced visual features provide more accurate input for subsequent grasping operations, thereby improving the robotic arm's operational precision and flexibility in various environments.

[0146] In one embodiment, the above step S40 includes:

[0147] S401, combining the visual features and the semantic representation by an action generation module to form a fusion feature;

[0148] S402, predicting the three-dimensional position coordinates and posture angle of the grasping point based on the fusion features by the action generation module;

[0149] S403, planning a motion trajectory of the robotic arm from the current position to the grasping point based on the three-dimensional position coordinates, posture angle, and the current state of the robotic arm through the motion generation module;

[0150] S404, generating a closing force parameter of the gripper based on the attribute information of the target object in the fusion feature by the action generation module;

[0151] S405: Encode the motion trajectory and the closing force parameters into an action sequence through the action generation module.

[0152] In this embodiment, the action generation module first combines visual features and semantic representations to form fused features. Visual features represent the appearance of the target object, while semantic representations include the target object's operational intent and environmental context. This fusion of the two helps combine the object's physical properties with its intentional description, allowing subsequent action generation to take more factors into account, thereby improving the accuracy and flexibility of the action.

[0153] Next, the action generation module predicts the 3D position coordinates and posture angle of the grasping point based on the fused features. The 3D position coordinates represent the target object's position in space, while the posture angle represents its orientation. Based on the fused features, the action generation module can accurately determine the spatial position and angle of the grasping point, ensuring accurate grasping, especially for complex or irregular objects.

[0154] The motion generation module then uses the predicted 3D position coordinates and posture angles, along with the current state of the robotic arm, to plan the robotic arm's trajectory from its current position to the grasping point. This process takes into account factors such as the robotic arm's reachability, path planning, and obstacle avoidance, ensuring that the robotic arm reaches the target grasping position smoothly and accurately while following the optimal motion path. Trajectory planning is a critical step in ensuring that the robotic arm avoids collisions or errors during grasping.

[0155] In addition, the action generation module also generates the closing force parameters of the gripper based on the attribute information of the target object in the fusion feature. The attributes of the target object (such as the material, size, shape, etc.) determine the amount of force the gripper needs to apply. For example, for fragile objects, the gripper's force needs to be gentler, while for harder objects, the gripper's force can be increased. The implementation of this step enables the robotic arm to intelligently adjust according to the different characteristics of the object when performing the grasping operation.

[0156] Finally, the action generation module encodes the motion trajectory and closing force parameters into an action sequence. This action sequence encompasses every detail of the robot's entire grasping operation, from the motion trajectory to the gripper's force adjustment. Each action is encoded in detail to ensure that every step of the grasping operation is executed precisely. This process not only ensures the accuracy of the robot's movements but also improves grasping efficiency and reduces unnecessary repetitive operations.

[0157] In practical applications, the specific implementation can be adjusted to suit different scenarios. For example, in industrial automation, a robotic arm can dynamically adjust to objects of varying shapes and materials based on the actual needs of the production line. By continuously collecting visual and semantic information about the target object, the robotic arm can update the parameters of the action generation module in real time, allowing it to better adapt to the ever-changing production environment.

[0158] In healthcare, robotic arms can precisely control the trajectory and force of surgical tools by combining voice commands with visual information. For example, during minimally invasive surgery, robotic arms need to adjust their gripping force based on the patient's position and tissue type to avoid damaging delicate tissue.

[0159] In the financial sector, this technology can be applied to automated file management and warehouse item identification. By identifying the image features of an object and using voice commands, the robotic arm can automatically adjust its gripping method based on the object's attributes, such as extracting and accessing documents, metal objects, or other financial-related items.

[0160] This embodiment uses this method to achieve high-precision grasping operations. The fusion of visual features and semantic representations enables the robotic arm to not only identify the target object's form but also understand its operational intent, thereby generating more precise action sequences. Through effective path planning and gripping force adjustment, the robotic arm can adapt to various complex grasping tasks in different environments, reducing misoperations or failures caused by environmental changes, thereby improving the reliability and intelligence of the robotic arm.

[0161] In one embodiment, the above step S50 includes:

[0162] S501, parsing the motion trajectory and grasping posture parameters from the action sequence;

[0163] S502, loading the link parameters and joint constraint range of the robotic arm;

[0164] S503, solving the inverse kinematics equation based on the poses of the waypoints in the motion trajectory, in combination with the link parameters and the joint constraint range, to obtain a sequence of joint angles of the waypoints;

[0165] S504, verifying whether the path point joint angle sequence interferes with environmental obstacles;

[0166] S505, when interference with environmental obstacles occurs, modifying the path point joint angle sequence through iterative adjustment processing to avoid interference, and generating a final joint angle sequence;

[0167] S506 , calculating the joint angle value of the grasping point based on the position coordinates and posture angle in the grasping posture parameters, in combination with the link parameters and the joint constraint range.

[0168] In this embodiment, the motion trajectory and grasping posture parameters are first parsed from the action sequence. The motion trajectory contains the target object's motion path information, that is, the target object's movement trajectory from the starting point to the grasping point. The grasping posture parameters provide the target object's precise position and posture information required during the grasping process. With this information, the robot arm can understand the target object's motion trajectory and the final target position and posture, which serves as the basis for subsequent operations.

[0169] Next, load the robot's link parameters and joint constraint ranges. Link parameters, including link length and link angle, are physical quantities that describe the geometric relationships between the robot's joints and determine the robot's range of motion. Joint constraint ranges refer to the angular range that each joint can achieve. These parameters limit the robot's degrees of freedom, ensuring that the robot does not move beyond its limits or in an unnatural manner.

[0170] After obtaining the motion trajectory and grasping pose parameters, link parameters, and joint constraint ranges, the inverse kinematics equations are solved based on the pathpoint poses in the motion trajectory, combined with the link parameters and joint constraint ranges. Inverse kinematics is the process of determining the required angles for each joint in the robotic arm. Through inverse kinematics calculations, the angle values ​​of each joint in the robotic arm can be determined, allowing the robotic arm's end effector to accurately reach the target pose. The pathpoint joint angle sequence is the set of joint angles at each pathpoint as the robotic arm moves along the path, providing the necessary parameters for precise control of the robotic arm.

[0171] To ensure that the calculated pathpoint joint angle sequence does not collide with obstacles in the environment during actual operation, the next step is to verify whether the pathpoint joint angle sequence interferes with environmental obstacles. The collision detection process checks whether each pathpoint of the robot arm intersects with the preset environmental obstacles. If interference occurs, it indicates that the current pathpoint is not suitable for continued execution, which may cause grasping failure or damage to the robot arm.

[0172] When the waypoint joint angle sequence interferes with an environmental obstacle, an iterative adjustment process modifies the waypoint joint angle sequence to avoid interference. This process may involve adjusting joint angle values ​​or changing the robot's path to ensure the robot can complete the grasping task safely and smoothly. Through multiple iterations, the resulting joint angle sequence should ensure that the robot's movement is smooth and safe, and it can accurately reach the target grasping position.

[0173] Finally, the grasping point joint angles are calculated based on the position coordinates and posture angles in the grasping pose parameters, combined with the link parameters and joint constraint ranges. These grasping point joint angles are the angles that each joint needs to achieve for the precise grasp position of the target object, calculated based on inverse kinematics. This calculation is a critical step in ensuring that the robot arm can achieve a precise grasping posture at the final grasping position. Unlike the angle sequence of the path points, the angle values ​​of the grasping points require extremely high precision to ensure that the gripper can accurately grasp the target object and perform the corresponding operation.

[0174] Example: Imagine a 6-DOF robotic arm tasked with performing precise grasping operations based on a given target position and posture. The goal is to calculate the joint angles at each path point, control the robotic arm's movement from its starting point to the target position, and successfully grasp the target object.

[0175] In practical applications, the target position and pose are typically acquired through a vision system or external input. For example, the 3D position coordinates of the target object are (X, Y, Z), and the pose information (such as Euler angles) is known. The pose (position and pose) of each waypoint is used as input in the action sequence.

[0176] For example, suppose the target pose of waypoint 1 is: position coordinates: (X1, Y1, Z1); attitude angle: (Roll1, Pitch1, Yaw1).

[0177] This means that the robot end effector must move to coordinates (X1, Y1, Z1) and its attitude angle must be adjusted to (Roll1, Pitch1, Yaw1) to complete the task at this waypoint.

[0178] Every robotic arm has a unique geometry. Calculating joint angles requires knowing the arm's link lengths, joint types, and the range of motion of each joint. For example, the first joint of a robotic arm might be a revolute joint, the second a swing joint, and so on.

[0179] For example: length of link 1: L1; length of link 2: L2; range of motion of joint 1: θ1min≤θ1≤θ1max; range of motion of joint 2: θ2min≤θ2≤θ2max.

[0180] These parameters are crucial to the inverse kinematics solution process because they define the range of motion of the robot arm and ensure that the robot's motion is feasible.

[0181] Given the target position and posture, the inverse kinematics algorithm reversely calculates the angle of each joint based on the robot's geometric structure. The core of this process is to solve the joint angles through the inverse kinematics equations, so that the robot's end effector can accurately reach the target position and have the correct posture.

[0182] Assuming the target position is (X1, Y1, Z1), the inverse kinematics equations can be used to calculate the angles of joint 1 (θ1) and joint 2 (θ2). The specific solution process can be solved using analytical methods (such as trigonometric functions) or numerical optimization methods (such as Newton's method or gradient descent).

[0183] For example, for a simple two-degree-of-freedom robotic arm, the joint angles can be calculated using the following relationship:

[0184] θ1=atan2(Y1,X1);

[0185] θ2=acos((X1 2 +Y1 2 -L1 2 -L2 2 ) / (2*L1*L2)).

[0186] Through these formulas, the angle value of each joint can be obtained.

[0187] During robotic arm motion, collision detection is an important step in ensuring safe and reliable operation. Suppose, at certain path points, the robotic arm's end effector collides with an obstacle in the environment. In this case, the joint angles calculated by the inverse kinematics solution need to be adjusted to avoid the collision.

[0188] For example, at path point 2, collision detection reveals that the end of the robot arm has contacted an object. To address this, iterative optimization can be used to adjust the joint angles until the path no longer interferes with the obstacle. This adjustment is typically achieved using numerical optimization methods, such as gradient descent, to optimize the path so that the robot arm avoids the obstacle while continuing to move along the target path.

[0189] Once collision detection is passed and the joint angles are adjusted, the final joint angle sequence can be generated. These joint angles are the control parameters for the robot to execute the motion trajectory, which determines the position and posture of the robot at each waypoint.

[0190] For example, the sequence of joint angles for a robotic arm might be:

[0191] θ1=45°; θ2=30°.

[0192] These angles control how the robotic arm moves, ensuring it can precisely reach its target location and complete its task.

[0193] This embodiment uses inverse kinematics calculations and environmental collision avoidance to ensure the robotic arm can safely and accurately grasp the target object. By introducing iterative adjustments and path optimization, the robotic arm's flexibility and safety are guaranteed, while also ensuring high-precision control of the grasping process.

[0194] In one embodiment, the above step S60 includes:

[0195] S601, sending the joint angle sequence and joint angle values ​​to the joint controller of the robotic arm;

[0196] S602, controlling the robotic arm to move according to the joint angle sequence through the joint controller;

[0197] S603, when the robotic arm reaches the end point of the motion trajectory, adjusting the robotic arm joint to a position corresponding to the joint angle value;

[0198] S604, controlling the gripper of the robotic arm to close to grasp the target object based on the position coordinates and posture angle corresponding to the joint angle value;

[0199] S605, detecting whether the target object is successfully grasped;

[0200] S606, when the target object is successfully grasped, controlling the gripper to maintain a closed state;

[0201] S607: When the target object is not successfully grasped, re-plan the action sequence and perform the grasping operation.

[0202] In this embodiment, the joint angle sequence and joint angle values ​​must first be sent to the joint controller. The joint angle sequence is a series of joint angle values ​​obtained through the previous inverse kinematics calculation, representing the path points of the robot arm from the starting position to the target position. The joint angle value is the precise joint angle at the target position, representing the final angle of the robot arm after completing the path movement. This information needs to be accurately transmitted to the joint controller, which is responsible for converting these control commands into specific movements of the robot arm.

[0203] After receiving this data, the joint controller begins controlling the robot's joints according to the provided joint angle sequence. Specifically, the controller uses the angle information at each path point in the joint angle sequence to ensure smooth movement of the robot along the prescribed trajectory, ensuring that the robot avoids obstacles and minimizes collisions and path deviations as it transitions from its starting position to its target position.

[0204] When the robot reaches the end of its trajectory, the controller adjusts the arm's joints to the positions corresponding to the joint angle values. This process uses precisely calculated target angles to ensure the end effector reaches the target pose. At this point, the robot has completed continuous motion from its initial position to the target position and is ready to perform a precise grasping operation.

[0205] Based on the current position coordinates and attitude angle of the robot arm's end effector, the controller further instructs the gripper to close and grasp the target object. The gripper's closing force is typically adjusted based on the characteristics of the target object to ensure a secure grip and avoid damage or drop due to loose or excessive grip.

[0206] After the grasping operation is completed, the controller needs to detect whether the target object has been successfully grasped through sensors or visual feedback systems. This process involves the cooperation of sensors and feedback mechanisms. By detecting the closing state of the gripper and the displacement of the object, it is determined whether the target object has been firmly grasped. If the target object is successfully grasped, the controller will instruct the gripper to remain closed and continue to perform subsequent operations. If the target object is not successfully grasped, the controller will replan the action sequence, adjust the grasping strategy, and execute the grasping operation. This process ensures that in the event of failure, the robot arm can autonomously adjust and make a second grasping attempt, enhancing the robustness of the operation.

[0207] Example: In actual applications on industrial production lines, robotic arms are often used to grab different types of parts and place them precisely in designated locations. To achieve this, robotic arms need to be able to quickly and accurately locate and grab the target object. Taking an industrial robot grabbing a metal part as an example, first, the operator issues an operation request through a voice command to "grab the metal part on the right." At this point, the system receives the voice input and converts it into a natural language instruction in text form through the voice recognition module. The instruction is then input into the language processing module, which performs word segmentation processing, extracts the keywords "grab" and "metal part" in the instruction, and generates a processable semantic representation. Based on this, the system determines that the task goal is to grab the metal part on the right.

[0208] At the same time, the system uses optical cameras and depth sensors to obtain visual information about the target object's environment. Color images are used to provide basic information about the object's appearance, while depth sensors provide depth information about the spatial relationship between the object and its environment. This information is transmitted to the visual processing module, which processes this input data through a convolutional neural network to extract the geometric shape and texture features of the target object. The feature pyramid network further performs multi-scale processing on features at each level in the image to obtain the target object's precise position and posture information. After depth information processing, the final visual features are input into the action generation module together with the semantic representation generated by the language module, forming a complete understanding of the robotic arm's grasping task.

[0209] The action generation module fuses visual features with semantic representations to generate fused features that include information such as the target object's 3D position and posture angle. Based on these fused features, the module first predicts the 3D position coordinates and posture angle of the metal part and then plans the robot's motion trajectory from its current position to the grasping point. To ensure the accuracy of the grasping operation, the action generation module also calculates the required gripper closing force parameters based on the target object's attributes. Finally, the generated action sequence, including the motion trajectory and gripper closing force information, is sent to the robot's control system.

[0210] In the motion control of the robotic arm, action sequences are used to generate joint angle sequences and joint angle values. The joint angle sequence is used to plan the motion path of the robotic arm from the starting position to the target position, ensuring that the robotic arm moves smoothly according to the predetermined trajectory and avoids collisions with obstacles in the surrounding environment. The inverse kinematics solution process calculates the corresponding joint angle sequence based on the posture of each path point and verifies whether the calculated path has any collision problems. If there is interference, the system will optimize and adjust the joint angle sequence of the path point to avoid the interference area until the final joint angle sequence is generated. At the same time, the joint angle value of the grasping point is obtained by further calculation of the grasping posture parameters to ensure that the robotic arm can accurately perform the grasping operation when it reaches the target position.

[0211] The robotic arm joint controller transmits the received joint angle sequence and values ​​to the robotic arm, and uses controller instructions to drive the robotic arm along the path sequence. When the robotic arm reaches the target position, the controller adjusts the robotic arm's joints to ensure that the end effector (gripper) is accurately aligned with the target object. The gripper then closes according to the calculated closing force parameters to ensure the appropriate force during the grasping process, avoiding damage to the object or grasping failure.

[0212] Finally, the system performs real-time detection to determine whether the target object has been successfully grasped. If the detection indicates a successful grasp, the gripper remains closed, completing the grasp. If the grasp is unsuccessful, the system automatically replans the motion sequence and re-executes the grasp operation to improve the grasp success rate and the robustness of the robotic arm system.

[0213] This process ensures that the robotic arm can perform grasping operations in dynamic and complex environments through highly precise motion control and real-time feedback, and can achieve efficient grasping under different conditions, whether it is stable metal parts or parts with complex shapes.

[0214] In healthcare, particularly in surgical robotics and patient care, precise grasping and manipulation by robotic arms are crucial. First, medical devices receive operational instructions through voice recognition technology. For example, a doctor might voice-command "grab a scalpel." This voice input is converted into text-based natural language instructions by a speech processing module. The language processing module extracts the key words "grab" and "scalpel" from the instruction and generates a corresponding semantic representation, indicating that the task objective is to grasp a scalpel.

[0215] Simultaneously, the operating room's camera system and depth sensors monitor the target object in real time, collecting visual information about the object's position, including the scalpel, and its surroundings. The color image provides basic information about the object's appearance, while the depth sensor accurately captures the distance and spatial relationship between the object and its surroundings. After receiving this information, the visual processing module uses a convolutional neural network to process the input data, extracting the scalpel's geometric features. It then uses a feature pyramid network to further analyze texture information at different scales, ultimately determining the precise object position and posture angle.

[0216] These processed visual features, along with the semantic representation generated by the language module, are input into the action generation module, forming a comprehensive understanding of the grasping task. Based on the fused visual and semantic information, the action generation module predicts the scalpel's three-dimensional position coordinates and posture angles and plans the robot's motion trajectory from its current position to the target position. To ensure that the scalpel is grasped at the correct time and location, the action generation module generates the required closing force parameters for the gripper based on the properties of the target object. Ultimately, all this information is combined into an action sequence, which is then transmitted to the robot's control system.

[0217] In robotic arm control, motion sequences are converted into joint angle sequences and joint angle values. The joint angle sequence ensures a smooth and collision-free path from the initial position to the target position, while the joint angle values ​​ensure that the gripper can accurately align with the target object during grasping. Through inverse kinematics calculations, each path point of the robotic arm is converted into a corresponding joint angle, which is then subjected to collision detection and optimized adjustments to ensure safe and efficient task execution. In particular, the joint angle values ​​of the grasping point are derived from the precise calculation of the scalpel's grasping position, ensuring that the gripper does not cause damage to the object during grasping.

[0218] As the robotic arm moves according to the calculated joint angle sequence, the joint controller precisely drives each joint according to the pre-set motion sequence. Upon reaching the target position, the joint controller adjusts the joint angles to ensure the gripper is aligned with the scalpel, ready for grasping. Based on the calculated closing force parameters, the gripper closes, ensuring a secure grip on the scalpel with the appropriate force.

[0219] Finally, the system detects the success of the grasping operation in real time. If the target object is successfully grasped, the gripper remains closed, completing the task. If the grasping fails, the system replans the action sequence based on pre-set logic, adjusting the grasping angle or force parameters, and continues the grasping operation until it succeeds.

[0220] In the healthcare sector, especially in precision surgical procedures, robotic arms can accurately and quickly perform grasping tasks, enabling efficient and reliable operation under diverse environmental conditions. This not only improves the precision of medical procedures but also reduces errors in human operation, significantly enhancing surgical safety and efficiency.

[0221] In the field of FinTech, particularly in intelligent financial robots and automated trading systems, robotic gripping and manipulation technologies can be applied in a variety of scenarios, such as asset management, risk assessment, and intelligent monitoring of capital flows. First, financial systems receive operational instructions through natural language processing. For example, users of asset management platforms can use voice input to enter commands, such as "Transfer funds to account 12345." This voice input is converted into text instructions by a speech recognition system. The language processing module extracts the core content of the instruction, identifies the "transfer funds" and the target account information, and generates a semantic representation. This semantic representation of the instruction provides guidance for subsequent action generation.

[0222] At the same time, visual information from financial data, such as signed documents, checks, or bills, is also captured in real time by image acquisition devices. The system uses cameras and depth sensors to acquire these images and depth data, identifying the document's content and spatial location. The visual processing module processes the input image, extracting the document's geometric and texture features to identify key information such as the bill's content, amount, and invoice date, and generates corresponding visual features.

[0223] These visual features, along with the semantic representations generated by the language module, are then passed to the action generation module, which generates corresponding action sequences based on actual application requirements. For example, in automated banking operations, the action generation module generates a series of actions based on the recognized account information and fund transfer requests, physically controlling the robot to perform bill grabbing and fund transfer operations.

[0224] After receiving the motion sequence, the system converts it into a joint angle sequence and joint angle values, ensuring the financial robot's smooth and precise operation path. The joint angle sequence controls the robot's motion trajectory, while the joint angle values ​​ensure precise execution of the final grasping operation, avoiding deviations during the transfer of financial documents or assets. Through inverse kinematics algorithms and collision detection, the robotic arm can move smoothly and complete fund transfers or bill processing tasks at precise locations.

[0225] The joint control system precisely controls the robot's movements based on the calculated joint angle sequence and values. As the robot approaches the target location, the system adjusts the position based on the joint angle values ​​to ensure a successful grasping operation. Once the target object is grasped, the robot maintains its grip and completes the transfer of the document or asset. The system also monitors the grasping status based on real-time feedback, ensuring accuracy before continuing with subsequent tasks.

[0226] In the field of financial technology, it can be widely applied to tasks such as automated asset management, paperless transactions, and intelligent investment. It not only improves operational accuracy but also reduces human error, significantly enhancing operational efficiency and security. In intelligent asset transfers, the system can rapidly process the flow of funds and assets while ensuring high-precision capture and risk control, ensuring the real-time and security of financial transactions. These technologies can provide the financial industry with more efficient workflows and automated processing capabilities, driving the further intelligence and modernization of financial services.

[0227] This embodiment can effectively improve the grasping accuracy of the robotic arm in dynamic and complex environments by precisely controlling the motion trajectory and terminal posture of the robotic arm. The continuity of the joint angle sequence ensures a smooth transition of the robotic arm on the path, avoids sudden changes in the motion trajectory, thereby reducing the risk of collision and improving safety. The precise adjustment of the joint angle value ensures that the end effector of the robotic arm can be accurately positioned to the target object, ensuring high precision and stability during the grasping process. Through a real-time feedback mechanism, the system can automatically adjust and re-plan the action sequence when the grasping fails, thereby increasing the success rate of the grasping and the robustness of the operation.

[0228] In one embodiment, a target manipulation device based on multimodal information is provided, and the target manipulation device based on multimodal information corresponds one-to-one to the target manipulation method based on multimodal information in the above embodiment. Figure 3 , Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of a multimodal information-based target manipulation device of the present invention. It includes a sensor module 10, a language processing module 20, a visual processing module 30, an action generation module 40, a kinematics calculation module 50, and a control module 60. Each functional module is described in detail below:

[0229] The sensor module 10 is used to obtain language instructions expressing the operation intention and collect visual information of the environment where the target object is located;

[0230] A language processing module 20, configured to input the language instruction into a language processing module and generate a semantic representation through the language processing module;

[0231] A visual processing module 30, configured to input the visual information into a visual processing module and generate visual features through the visual processing module;

[0232] An action generation module 40 is configured to input the visual features and the semantic representation into an action generation module, and generate an action sequence through the action generation module;

[0233] a kinematics calculation module 50, configured to determine a joint angle sequence and joint angle values ​​based on the motion sequence;

[0234] The control module 60 is configured to control the robotic arm based on the joint angle sequence and the joint angle values ​​to perform a grasping operation on the target object.

[0235] In one embodiment, the sensor module 10 is specifically configured to:

[0236] receiving a voice input for expressing an operation intention, and converting the voice input into a natural language instruction in text form;

[0237] The optical camera collects a color image of the environment where the target object is located, and the depth sensor collects depth information of the environment where the target object is located;

[0238] Converting the depth information into three-dimensional point cloud spatial coordinate data, and performing outlier filtering on the three-dimensional point cloud spatial coordinate data;

[0239] The three-dimensional point cloud spatial coordinate data after outlier filtering is aligned with the color image to generate the visual information.

[0240] In one embodiment, the language processing module 20 is specifically configured to:

[0241] Performing word segmentation processing on the language instruction by a language processing module to generate a plurality of word units;

[0242] Convert each word unit into a word embedding vector through the language processing module to obtain a word embedding vector sequence;

[0243] Processing the word embedding vector sequence through a recurrent neural network in the language processing module to generate contextual features;

[0244] The context features are processed by a language model in the language processing module to generate a semantic representation.

[0245] In one embodiment, the visual processing module 30 is specifically configured to:

[0246] Separating a mask region of the target object from the visual information by a visual processing module;

[0247] Processing the mask area through a convolutional neural network in the visual processing module to obtain geometric shape features;

[0248] Processing the mask area through a feature pyramid network in the visual processing module to obtain multi-scale texture features;

[0249] The geometric shape features and the multi-scale texture features are fused through the visual processing module, and visual features are generated based on the fused features.

[0250] In one embodiment, the action generation module 40 is specifically configured to:

[0251] splicing the visual features and the semantic representation through an action generation module to form a fusion feature;

[0252] Predicting the three-dimensional position coordinates and posture angle of the grasping point based on the fusion features by the action generation module;

[0253] Planning, by the action generation module, a motion trajectory of the robotic arm from the current position to the grasping point based on the three-dimensional position coordinates, the posture angle, and the current state of the robotic arm;

[0254] generating, by the action generation module, a closing force parameter of the gripper based on the attribute information of the target object in the fusion feature;

[0255] The motion trajectory and the closing force parameters are encoded into an action sequence through the action generation module.

[0256] In one embodiment, the kinematics calculation module 50 is specifically configured to:

[0257] parsing motion trajectory and grasping posture parameters from the action sequence;

[0258] Load the link parameters and joint constraint range of the robotic arm;

[0259] Based on the poses of the waypoints in the motion trajectory, combined with the link parameters and joint constraint ranges, the inverse kinematics equations are solved to obtain a sequence of waypoint joint angles;

[0260] Verify whether the path point joint angle sequence interferes with environmental obstacles;

[0261] When interference with environmental obstacles occurs, the path point joint angle sequence is modified through iterative adjustment processing to avoid the interference and generate a final joint angle sequence;

[0262] Based on the position coordinates and posture angles in the grasping posture parameters, combined with the link parameters and joint constraint range, the grasping point joint angle value is solved.

[0263] In one embodiment, the control module 60 is specifically configured to:

[0264] Sending the joint angle sequence and the joint angle values ​​to a joint controller of the robotic arm;

[0265] Controlling the robotic arm to move according to the joint angle sequence through the joint controller;

[0266] When the robotic arm reaches the end point of the motion trajectory, the robotic arm joint is adjusted to a position corresponding to the joint angle value;

[0267] Based on the position coordinates and posture angles corresponding to the joint angle values, controlling the gripper of the robotic arm to close to grasp the target object;

[0268] Detecting whether the target object is successfully grasped;

[0269] When the target object is successfully grasped, controlling the gripper to maintain a closed state;

[0270] When the target object is not grasped successfully, the action sequence is replanned and the grasping operation is performed.

[0271] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide determination and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external user terminal via a network connection. When the computer program is executed by the processor, it realizes the functions or steps on the server side of a target manipulation method based on multimodal information.

[0272] In one embodiment, a computer device is provided. The computer device may be a user terminal, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, memory, network interface, display screen and input device connected via a system bus. Among them, the processor of the computer device is used to provide determination and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it realizes the functions or steps on the user side of a target control method based on multimodal information.

[0273] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are performed:

[0274] Obtain language instructions that express operational intentions and collect visual information about the environment where the target object is located;

[0275] inputting the language instruction into a language processing module, and generating a semantic representation through the language processing module;

[0276] Inputting the visual information into a visual processing module, and generating visual features through the visual processing module;

[0277] Inputting the visual features and the semantic representation into an action generation module, and generating an action sequence through the action generation module;

[0278] determining a joint angle sequence and joint angle values ​​based on the action sequence;

[0279] The robotic arm is controlled based on the joint angle sequence and the joint angle values ​​to perform a grasping operation on the target object.

[0280] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0281] Obtain language instructions that express operational intentions and collect visual information about the environment where the target object is located;

[0282] inputting the language instruction into a language processing module, and generating a semantic representation through the language processing module;

[0283] Inputting the visual information into a visual processing module, and generating visual features through the visual processing module;

[0284] Inputting the visual features and the semantic representation into an action generation module, and generating an action sequence through the action generation module;

[0285] determining a joint angle sequence and joint angle values ​​based on the action sequence;

[0286] The robotic arm is controlled based on the joint angle sequence and the joint angle values ​​to perform a grasping operation on the target object.

[0287] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can be found in the relevant descriptions of the server side and the user side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0288] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0289] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0290] It should be noted that if any software tools or components other than those of the Company appear in the embodiments of this application, they are merely for illustration and do not represent actual use. The above embodiments are intended only to illustrate the technical solutions of the present invention, not to limit them. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some of the technical features therein with equivalents. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. A target manipulation method based on multimodal information, characterized in that: The following steps are involved: Obtain language instructions that express operational intentions and collect visual information about the environment where the target object is located; inputting the language instruction into a language processing module, and generating a semantic representation through the language processing module; Inputting the visual information into a visual processing module, and generating visual features through the visual processing module; Inputting the visual features and the semantic representation into an action generation module, and generating an action sequence through the action generation module; determining a joint angle sequence and joint angle values ​​based on the action sequence; The robotic arm is controlled based on the joint angle sequence and the joint angle values ​​to perform a grasping operation on the target object.

2. The target manipulation method based on multimodal information according to claim 1, characterized in that: Obtain language instructions that express operational intent and collect visual information about the target object's environment, including: receiving a voice input for expressing an operation intention, and converting the voice input into a natural language instruction in text form; The optical camera collects a color image of the environment where the target object is located, and the depth sensor collects depth information of the environment where the target object is located; Converting the depth information into three-dimensional point cloud spatial coordinate data, and performing outlier filtering on the three-dimensional point cloud spatial coordinate data; The three-dimensional point cloud spatial coordinate data after outlier filtering is aligned with the color image to generate the visual information.

3. The target manipulation method based on multimodal information according to claim 1, characterized in that: Inputting the language instruction into a language processing module, and generating a semantic representation through the language processing module, including: Performing word segmentation processing on the language instruction by a language processing module to generate a plurality of word units; Convert each word unit into a word embedding vector through the language processing module to obtain a word embedding vector sequence; Processing the word embedding vector sequence through a recurrent neural network in the language processing module to generate contextual features; The context features are processed by a language model in the language processing module to generate a semantic representation.

4. The target manipulation method based on multimodal information according to claim 1, characterized in that: Inputting the visual information into a visual processing module, and generating visual features through the visual processing module, including: Separating a mask region of the target object from the visual information by a visual processing module; Processing the mask area through a convolutional neural network in the visual processing module to obtain geometric shape features; Processing the mask area through a feature pyramid network in the visual processing module to obtain multi-scale texture features; The geometric shape features and the multi-scale texture features are fused through the visual processing module, and visual features are generated based on the fused features.

5. The target manipulation method based on multimodal information according to claim 1, characterized in that: Inputting the visual features and the semantic representation into an action generation module, and generating an action sequence through the action generation module, comprising: splicing the visual features and the semantic representation through an action generation module to form a fusion feature; Predicting the three-dimensional position coordinates and posture angle of the grasping point based on the fusion features by the action generation module; Planning, by the action generation module, a motion trajectory of the robotic arm from the current position to the grasping point based on the three-dimensional position coordinates, the posture angle, and the current state of the robotic arm; generating, by the action generation module, a closing force parameter of the gripper based on the attribute information of the target object in the fusion feature; The motion trajectory and the closing force parameters are encoded into an action sequence through the action generation module.

6. The target manipulation method based on multimodal information according to claim 1, characterized in that: Determining a joint angle sequence based on the action sequence includes: parsing motion trajectory and grasping posture parameters from the action sequence; Load the link parameters and joint constraint range of the robotic arm; Based on the poses of the waypoints in the motion trajectory, combined with the link parameters and joint constraint ranges, the inverse kinematics equations are solved to obtain a sequence of waypoint joint angles; Verify whether the path point joint angle sequence interferes with environmental obstacles; When interference with environmental obstacles occurs, the path point joint angle sequence is modified through iterative adjustment processing to avoid the interference and generate a final joint angle sequence; Based on the position coordinates and posture angles in the grasping posture parameters, combined with the link parameters and joint constraint range, the grasping point joint angle value is solved.

7. The target manipulation method based on multimodal information according to claim 1, characterized in that: Controlling the robotic arm based on the joint angle sequence and the joint angle values ​​to perform a grasping operation on the target object includes: Sending the joint angle sequence and the joint angle values ​​to a joint controller of the robotic arm; Controlling the robotic arm to move according to the joint angle sequence through the joint controller; When the robotic arm reaches the end point of the motion trajectory, the robotic arm joint is adjusted to a position corresponding to the joint angle value; Based on the position coordinates and posture angles corresponding to the joint angle values, controlling the gripper of the robotic arm to close to grasp the target object; Detecting whether the target object is successfully grasped; When the target object is successfully grasped, controlling the gripper to maintain a closed state; When the target object is not grasped successfully, the action sequence is replanned and the grasping operation is performed.

8. A target manipulation device based on multimodal information, characterized in that: The target manipulation device based on multimodal information includes: The sensor module is used to obtain language instructions that express the operation intention and collect visual information about the environment where the target object is located; a language processing module, configured to input the language instruction into the language processing module and generate a semantic representation through the language processing module; a visual processing module, configured to input the visual information into the visual processing module and generate visual features through the visual processing module; An action generation module, configured to input the visual features and the semantic representation into the action generation module, and generate an action sequence through the action generation module; a kinematics calculation module, configured to determine a joint angle sequence and joint angle values ​​based on the motion sequence; A control module is used to control the robotic arm based on the joint angle sequence and the joint angle value to perform a grasping operation on the target object.

9. A computer device, characterized in that: The computer device includes a memory, a processor, and a target manipulation program based on multimodal information stored in the memory and capable of running on the processor. When the target manipulation program based on multimodal information is executed by the processor, the steps of the target manipulation method based on multimodal information are implemented as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The storage medium stores a target manipulation program based on multimodal information, and when the target manipulation program based on multimodal information is executed by the processor, the steps of the target manipulation method based on multimodal information are implemented as described in any one of claims 1 to 7.

Images