Action reasoning method and device for immune injection

Abstract

The invention provides an action reasoning method and device for immune injection, and relates to the technical field of computers. The method comprises the following steps: determining position and posture information of an injector and position and posture information of a target to be injected according to a current injection image; the injector pose information and the to-be-injected target pose information are input into a pre-trained multi-mode large model LLaVA, an action sequence corresponding to the injector pose information and the to-be-injected target pose information is obtained, and the action sequence comprises a plurality of actions; the multi-modal large model LLaVA understands the content accumulated by multiple rounds of dialogues on the teaching injection video through a visual question and answer technology, and learns to obtain the mapping relation between the position and posture information of an injector and the position and posture information of a target to be injected and an action sequence; and the mechanical arm is controlled to execute actions in the action sequence. According to the method, the action sequence suitable for the current scene is obtained by combining the multi-mode large model LLaVA with imitation learning, the method can adapt to individual differences of different environments and to-be-injected targets, and the efficiency and accuracy of vaccine injection are improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, in particular to the field of intelligent animal husbandry technology, and in particular to a motion reasoning method and device for immunization injection. Background Art

[0002] In modern animal husbandry, vaccinating livestock is a key operation that is directly related to the health of livestock and the benefits of breeding. Traditionally, this task is done manually by veterinarians or experienced staff, but this method is labor-intensive and has low injection efficiency in large-scale farms. It is difficult to ensure that every livestock can be vaccinated in a timely and accurate manner, which affects production efficiency and increases the safety risks of operators. With the advancement of robotics technology, it has become feasible and necessary to try to use automated robots to complete vaccination tasks. However, most of the current vaccine injection robots do not have the ability to adjust themselves autonomously. The operating parameters and action modes of the injection robots are fixed, and they cannot adapt to different environments and individual differences of livestock, which affects the efficiency and accuracy of the injection. Summary of the invention

[0003] The present disclosure aims to solve one of the technical problems in the related art at least to some extent.

[0004] To this end, the first embodiment of the present disclosure proposes an action reasoning method for immune injection, the method is applied to an injection robot, the injection robot controls the action of the syringe through a mechanical arm, and the method comprises the following steps:

[0005] Acquire a current injection image, wherein the current injection image includes a syringe and a target to be injected;

[0006] Determine the syringe position information and the target position information to be injected according to the current injection image;

[0007] The syringe posture information and the target posture information to be injected are input into a pre-trained multimodal large model LLaVA to obtain an action sequence corresponding to the syringe posture information and the target posture information to be injected, wherein the action sequence includes a plurality of actions; the multimodal large model LLaVA uses visual question-answering technology to conduct a multi-round dialogue on the teaching injection video to accumulate content understanding, and learns to obtain a mapping relationship between the syringe posture information, the target posture information to be injected, and the action sequence;

[0008] Control the robot arm to execute the actions in the action sequence.

[0009] In some embodiments of the present disclosure, the controlling the robotic arm to perform the actions in the action sequence includes: S1, controlling the robotic arm to perform the first N actions in the action sequence to obtain an injection image after performing the action; wherein N is a positive integer; S2, using the posture information of the syringe in the injection image after performing the action as new syringe posture information, using the posture information of the target to be injected in the injection image after performing the action as new target posture information to be injected, inputting the new syringe posture information and the new target posture information to be injected into the multimodal large model LLaVA to obtain a new action sequence; S3, controlling the robotic arm to perform the first N actions in the new action sequence; S4, repeating steps S2-S3 until the robotic arm performs a preset injection end action.

[0010] In some embodiments of the present disclosure, the multimodal large model LLaVA uses visual question-answering technology to conduct multiple rounds of dialogue on the teaching injection video to accumulate content understanding, and learn to obtain the mapping relationship between the syringe posture information, the target posture information to be injected, and the action sequence, including: segmenting and sampling the teaching injection video to obtain continuous video frame images; determining the initial syringe posture information of the syringe and the initial target posture information to be injected of the target to be injected in the teaching injection video; the multimodal large model LLaVA uses visual question-answering technology to select the initial action in the continuous video frame image from multiple injection actions in the action skill library through multiple rounds of dialogue tasks, and generates an initial action sequence corresponding to the initial syringe posture information and the initial target posture information to be injected based on the initial action; obtains the action sequence label in the teaching injection video, and the action sequence label is determined based on the robot vaccine injection operation skill knowledge; the multimodal large model LLaVA is trained based on the action sequence label and the initial action sequence, so that the multimodal large model LLaVA learns to obtain the mapping relationship between the syringe posture information and the target posture information to be injected and the action sequence.

[0011] In some embodiments of the present disclosure, the syringe posture information includes position information and spatial posture information of the syringe, and the spatial posture information of the syringe is acquired in the following manner: establishing a 3D model of the syringe; using a deep learning-based visual algorithm to extract feature information of the syringe in the current injection image; matching the feature information with the 3D model to capture the spatial posture information of the syringe.

[0012] In some embodiments of the present disclosure, the action includes an action type and an action parameter.

[0013] The second aspect of the present disclosure provides a motion reasoning device for immunization injection, the device is used to control the motion of the syringe through the mechanical arm of the injection robot, and the device includes:

[0014] An acquisition module, used for acquiring a current injection image, wherein the current injection image includes a syringe and a target to be injected;

[0015] A first determination module, used to determine the syringe posture information and the to-be-injected target posture information according to the current injection image;

[0016] The second determination module is used to input the syringe posture information and the target posture information to be injected into a pre-trained multimodal large model LLaVA to obtain an action sequence corresponding to the syringe posture information and the target posture information to be injected, wherein the action sequence includes a plurality of actions; the multimodal large model LLaVA uses visual question-answering technology to conduct a multi-round dialogue on the teaching injection video to accumulate content understanding, and learn to obtain a mapping relationship between the syringe posture information, the target posture information to be injected, and the action sequence;

[0017] A control module is used to control the robot arm to execute the actions in the action sequence.

[0018] In some embodiments of the present disclosure, the control module is specifically used for: S1, controlling the robotic arm to execute the first N actions in the action sequence to obtain an injection image after executing the action; wherein, N is a positive integer; S2, using the posture information of the syringe in the injection image after executing the action as new syringe posture information, using the posture information of the target to be injected in the injection image after executing the action as new target posture information to be injected, inputting the new syringe posture information and the new target posture information to be injected into the multimodal large model LLaVA to obtain a new action sequence; S3, controlling the robotic arm to execute the first N actions in the new action sequence; S4, repeating steps S2-S3 until the robotic arm executes a preset injection end action.

[0019] In some embodiments of the present disclosure, the device also includes a training device; the training device is used to: segment and sample the teaching injection video to obtain continuous video frame images; determine the initial syringe posture information of the syringe in the teaching injection video and the initial target posture information of the target to be injected; the multimodal large model LLaVA adopts visual question-answering technology to select the initial action in the continuous video frame image from multiple injection actions in the action skill library through multiple rounds of dialogue tasks, and generates an initial action sequence corresponding to the initial syringe posture information and the initial target posture information to be injected based on the initial action; obtain the action sequence label in the teaching injection video, and the action sequence label is determined based on the robot vaccine injection operation skill knowledge; train the multimodal large model LLaVA based on the action sequence label and the initial action sequence, so that the multimodal large model LLaVA learns to obtain the mapping relationship between the syringe posture information and the target posture information to be injected and the action sequence.

[0020] A third aspect of the present disclosure provides an electronic device, comprising: a processor, and a memory communicatively connected to the processor;

[0021] The memory stores computer-executable instructions;

[0022] The processor executes the computer-executable instructions stored in the memory to implement the method described in the first aspect.

[0023] The fourth aspect of the present disclosure provides a computer-readable storage medium, characterized in that the computer-readable storage medium stores computer execution instructions, and the computer execution instructions are used to implement the method described in the first aspect when executed by a processor.

[0024] The motion reasoning method for immunization provided by the present disclosure obtains the motion sequence of the syringe posture information and the target posture information in the current scene through the pre-trained multimodal large model LLaVA, and guides the mechanical arm of the injection robot to autonomously complete the vaccine injection and autonomously adapt to different environments and individual differences of the target to be injected. The present disclosure can significantly improve the efficiency of vaccine injection, reduce the risk of manual operation, ensure that every livestock can be vaccinated in time, and improve the overall production efficiency and safety of commercial breeding.

[0025] Additional aspects and advantages of the present disclosure will be given in part in the following description and in part will be obvious from the following description or learned through practice of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The above and / or additional aspects and advantages of the present disclosure will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0027] Figure 1 A flowchart of an action reasoning method for immune injection provided by an embodiment of the present disclosure;

[0028] Figure 2 A schematic diagram of an injection image provided by an embodiment of the present disclosure;

[0029] Figure 3 A schematic diagram of obtaining the spatial posture information of a syringe through a 3D model of the syringe provided in an embodiment of the present disclosure;

[0030] Figure 4 An implementation process of controlling a robotic arm to perform actions in an action sequence provided by an embodiment of the present disclosure;

[0031] Figure 5 A flowchart of a training method for a multimodal large model LLaVA provided in an embodiment of the present disclosure;

[0032] Figure 6 A multimodal large model LLaVA action parsing framework diagram provided by an embodiment of the present disclosure;

[0033] Figure 7 A schematic diagram of an action sequence template provided by an embodiment of the present disclosure;

[0034] Figure 8 A schematic diagram of an action sequence output by a multimodal large model LLaVA after topology optimization provided by an embodiment of the present disclosure;

[0035] Fig. 9 A schematic diagram of topological optimization of actions based on an action sequence template provided in an embodiment of the present disclosure;

[0036] Fig.10 An action reasoning device for immunization injection provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0037] Embodiments of the present disclosure are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present disclosure, and should not be construed as limiting the present disclosure.

[0038] The present disclosure proposes an action reasoning method and device for immunization injection. Specifically, the action reasoning method and device for immunization injection according to an embodiment of the present disclosure are described below with reference to the accompanying drawings.

[0039] Figure 1The flowchart of a motion reasoning method for immune injection provided by an embodiment of the present disclosure is shown in FIG. 1 , which is applied to an injection robot that controls the motion of the syringe through a mechanical arm. Figure 1 As shown, the action reasoning method for immunization injection may include the following steps:

[0040] Step 101, obtaining a current injection image, wherein the current injection image includes a syringe and a target to be injected.

[0041] The injection image can be collected by the RGB-D camera on the injection robot. Taking pigs as an example, Figure 2 Schematic diagram of an injection image provided by an embodiment of the present disclosure. Figure 2 As shown, the injection image includes a syringe and a target to be injected (a live pig).

[0042] Step 102, determining the syringe posture information and the object posture information to be injected according to the current injection image.

[0043] The posture information includes position information and spatial posture information. Optionally, the posture information of the syringe and the posture information of the target to be injected can be obtained by plane target detection technology.

[0044] In the related art, the spatial posture information of the syringe is usually obtained by installing a posture detection sensor on the syringe, but the learning cost of this method is high. At the same time, the presence of the sensor will hinder the expert's teaching action to some extent, affecting the accuracy of the operation, and thus affecting the imitation learning effect. In order to improve the efficiency of posture detection, reduce the learning cost, and not affect the smoothness of the expert's teaching operation, in some embodiments of the present disclosure, the spatial posture information of the syringe can also be obtained by matching the 3D model of the syringe. Specifically, in one implementation, Figure 3 As shown in the figure, the syringe can be modeled by using the NeRF-based visual 3D reconstruction algorithm of Solidworks to generate a digital format file and establish a 3D model of the syringe ( Figure 3 A deep learning-based visual algorithm is used to analyze the current injection image ( Figure 3 The feature extraction and analysis is performed by the neural network, and the characteristic information of the syringe in the current injection image, such as the shape and contour of the syringe, is identified through the learning ability of the neural network. Then the characteristic information is matched with the 3D model of the syringe, thereby accurately obtaining the spatial posture information of the syringe in three-dimensional space ( Figure 3 (Part c).

[0045] Step 103, inputting the syringe posture information and the target posture information to be injected into the pre-trained multimodal large model LLaVA to obtain an action sequence corresponding to the syringe posture information and the target posture information to be injected, wherein the action sequence includes multiple actions.

[0046] The action includes an action type and an action parameter. As an example, the action type may include positioning, tilting, releasing, pressing, etc., and the action parameter may include the position of the action in space, the thrust of the liquid pressing action during injection, etc.

[0047] It should be noted that the multimodal large model LLaVA has accumulated content understanding of multiple rounds of dialogue in the teaching injection video through visual question-answering technology, imitated and learned the injection action in the teaching injection video, and enabled the model to learn the mapping relationship between the syringe posture information and the posture information of the target to be injected and the action sequence, so that the action sequence output by the model is more adapted to the syringe posture information and the posture information of the target to be injected, thereby improving the injection accuracy. For the training method of the multimodal large model LLaVA in this embodiment, please refer to the description in the subsequent embodiments of this disclosure, which will not be repeated here.

[0048] Step 104: Control the robot arm to execute actions in the action sequence.

[0049] By implementing the embodiments of the present disclosure, the pre-trained multimodal large model LLaVA is combined with imitation learning to obtain the action sequence of the syringe posture information and the target posture information to be injected in the current scene, and the mechanical arm of the injection robot is guided to autonomously complete the vaccine injection and autonomously adapt to different environments and individual differences of the target to be injected. The present disclosure can significantly improve the efficiency of vaccine injection, reduce the risk of manual operation, ensure that every livestock can be vaccinated in time, and improve the overall production efficiency and safety of commercial breeding.

[0050] In some embodiments of the present disclosure, in order to further improve the accuracy of the injection process, the action sequence can be dynamically updated and optimized based on the real-time posture during the injection process. In one implementation, Figure 4 The present disclosure provides an implementation process for controlling a robot arm to perform actions in an action sequence. Figure 4 As shown, the implementation process of controlling the robot arm to perform actions in the action sequence may include the following steps:

[0051] Step 401, control the robot arm to execute the first N actions in the action sequence, and obtain the injection image after the actions are executed, where N is a positive integer.

[0052] Step 402: The posture information of the syringe in the injection image after the action is performed is used as new syringe posture information, and the posture information of the target to be injected in the injection image after the action is performed is used as new target posture information to be injected. The new syringe posture information and the new target posture information to be injected are input into the multimodal large model LLaVA to obtain a new action sequence.

[0053] As an example, taking N as 1, after controlling the robot arm to perform the first action in the action sequence, the target to be injected may move slightly, and the position of the target to be injected may change. At this time, continuing the action that has not been executed in the action sequence to inject the target to be injected may result in poor injection. In order to improve the accuracy of the injection action, the posture information of the syringe and the target to be injected after the first action is executed is used as the new syringe posture information and the new target posture information, and the new syringe posture information and the new target posture information are re-input into the multimodal large model LLaVA to obtain a new action sequence.

[0054] Step 403, control the robotic arm to execute the first N actions in the new action sequence.

[0055] Step 404, repeat steps 402-403 until the robotic arm performs a preset injection end action.

[0056] On the basis of the new syringe posture information and the new posture information of the target to be injected, continue to execute the first N actions in the new action sequence, repeat step 402-step 403, and after executing the first N actions in the new action sequence, redefine the posture information of the syringe and the target to be injected after the action is executed as the new syringe posture information and the new posture information of the target to be injected, and update the action sequence in real time until the robot arm executes the preset injection end action. Among them, the injection end action can be set as the syringe withdrawal action after the syringe pressurizes the liquid. When the robot arm executes the withdrawal action, it can be regarded as completing the injection of the target to be injected, and the cycle ends.

[0057] Therefore, by implementing steps 401 to 404, the action sequence can be dynamically updated and optimized based on the real-time posture of the syringe and the target to be injected during the injection process, so that the actions performed by the robot arm are more precise, further improving the injection accuracy of the target to be injected.

[0058] Figure 5 The flowchart of a multi-modal large model LLaVA training method provided by the embodiment of the present disclosure is as follows. Figure 5 As shown, the training method of the multimodal large model LLaVA may include the following steps:

[0059] Step 501 , segment and sample the teaching injection video to obtain continuous video frame images.

[0060] Figure 6 This is a multi-modal large model LLaVA action parsing framework diagram provided by the embodiment of the present disclosure. Figure 6 As shown, the teaching injection video is segmented and sampled to obtain continuous video frame images. In one implementation, the teaching injection video can be segmented and sampled according to a preset interval. The pairing of adjacent frames creates a frame combination to retain the timing information of the action, laying the foundation for action recognition and analysis.

[0061] Step 502 , determining the initial syringe posture information of the syringe in the teaching injection video and the initial target posture information of the target object to be injected.

[0062] Optionally, the initial syringe pose information of the syringe in the teaching injection video and the initial target pose information of the target to be injected can be obtained by plane target detection technology. Alternatively, the initial syringe pose information of the syringe can be obtained by matching the 3D model of the syringe. For specific acquisition methods, please refer to Figure 1 The relevant description of the embodiments will not be repeated here.

[0063] Step 503, the multimodal large model LLaVA adopts visual question answering technology to select the initial action in the continuous video frame images from multiple injection actions in the action skill library through multiple rounds of dialogue tasks, and generates an initial action sequence corresponding to the initial syringe posture information and the initial target posture information to be injected based on the initial action.

[0064] like Figure 6 As shown, the multimodal large model LLaVA combines visual and language processing capabilities, responds to natural language questions through visual question answering technology, and deeply analyzes visual content. ChatGPT plays the role of questioner, guiding the model to focus on specific parts of the video frame, such as object contact or object movement. Each difference between video frame images represents the changes between consecutive frames, which are the key basis for action detection and recognition. Through multiple rounds of dialogue, ChatGPT accumulates its understanding of the video content, selects corresponding actions from the predefined action skill library as the initial actions in the continuous video frame images, and then constructs a coherent initial action sequence corresponding to the initial syringe posture information and the initial target posture information to be injected. In addition, ChatGPT can also provide a language description containing context for the selected initial action to learn the association between actions and guide the model to complete complex tasks.

[0065] Step 504, obtaining the action sequence labels in the teaching injection video, where the action sequence labels are determined based on the robot vaccine injection operation skill knowledge.

[0066] It should be noted that the action sequence labels in the teaching injection video can be manually defined, and the robot vaccine injection operation skill knowledge includes the action sequence template in the robot's injection action. The action sequence template is different from the manual injection action. It is an action sequence suitable for the injection robot to perform by imitating and learning the human injection action. Figure 7 Schematic diagram of an action sequence template provided by an embodiment of the present disclosure. Figure 7 As shown, the instructor can refer to the action sequence template to define the action label of each action in the teaching injection process, and generate the action sequence label in the teaching injection video according to the action labels of multiple actions.

[0067] Step 505 , training the multimodal large model LLaVA based on the action sequence label and the initial action sequence, so that the multimodal large model LLaVA learns to obtain the mapping relationship between the syringe posture information and the target posture information to be injected and the action sequence.

[0068] The multimodal large model LLaVA is trained based on action sequence labels and initial action sequences. The initial action sequence output by the multimodal large model LLaVA is trained and corrected using the action sequence labels, so that the initial action sequence output by the multimodal large model LLaVA is more consistent with the action sequence template.

[0069] It should be noted that the action sequence template is not fully adapted to all situations. In the case of different syringe postures and the postures of the target to be injected, the action sequence template may contain redundant or useless actions. Therefore, the multimodal large model LLaVA uses visual question-answering technology to conduct multiple rounds of dialogue on the teaching injection video to accumulate content understanding, so that the multimodal large model LLaVA learns the association or topological relationship between each action in the action sequence, and obtains the ability to reason about the actions before and after (for example, through training, the large model learns through knowledge reasoning that there is a before and after association between the syringe liquid pressing completion action and the syringe withdrawal action). Therefore, the multimodal large model LLaVA can further topologically optimize the action based on the output that is more in line with the action sequence template according to the syringe posture information and the target posture information to be injected, and output the action sequence under the syringe posture information and the target posture information to be injected based on the action reasoning ability. Figure 8 A schematic diagram of an action sequence output by a multimodal large model LLaVA after topology optimization provided by an embodiment of the present disclosure.

[0070] The multimodal large model LLaVA is based on the learning and action reasoning capabilities of the action sequence template, so as to learn the mapping relationship between the syringe posture information and the position information of the target to be injected and the action sequence, so that the action sequence output by the model is more adapted to the syringe posture information and the position information of the target to be injected on the basis of fitting the action sequence template. The action sequence output by the trained multimodal large model LLaVA guides the injection robot to complete the autonomous injection of the vaccine, greatly improving the accuracy and adaptability of the vaccine injection, and significantly improving the injection efficiency and injection safety.

[0071] Optionally, in some embodiments of the present disclosure, Fig. 9 A schematic diagram of topological optimization of actions based on action sequence templates provided in an embodiment of the present disclosure. Fig. 9 As shown in FIG. 1 , the key frame information, spatial posture information, force information and position information of the scene can be obtained through the vaccine injection scene information. Among them, the key frame information is used to characterize the action stage of the syringe (such as Figure 8 The multimodal large model LLaVA can perform action reasoning and topology optimization based on the action sequence template according to the key frame information, spatial information, force information and posture information of the current injection scene, thereby outputting an action sequence that adapts to the current scene.

[0072] By implementing the disclosed embodiments, the multimodal large model LLaVA uses visual question-answering technology to imitate and train the injection actions in the teaching injection video, and learns the mapping relationship between the syringe posture information and the target posture information to be injected and the action sequence. The trained multimodal large model LLaVA can infer the injection action sequence under different injection scenarios in actual application scenarios, and autonomously adapt to different injection environments and individual injection differences, thereby improving the accuracy and adaptability of vaccine injection, significantly improving the efficiency of vaccine injection, making vaccination in large-scale commercial farming faster and more efficient, reducing the risk of contact for operators, and improving workplace safety.

[0073] Fig.10 The present disclosure provides an action reasoning device for immunization injection, which is used to control the action of the syringe through the mechanical arm of the injection robot. Fig.10 As shown, the motion reasoning device for immunization injection includes: an acquisition module 1001, a first determination module 1002, a second determination module 1003 and a control module 1004.

[0074] The acquisition module 1001 is used to acquire the current injection image, which includes the syringe and the target to be injected.

[0075] The first determination module 1002 is used to determine the syringe posture information and the object posture information to be injected according to the current injection image.

[0076] The second determination module 1003 is used to input the syringe posture information and the target posture information to be injected into the pre-trained multimodal large model LLaVA to obtain an action sequence corresponding to the syringe posture information and the target posture information to be injected, and the action sequence includes multiple actions. The multimodal large model LLaVA uses visual question-answering technology to conduct multiple rounds of dialogue and accumulate content understanding of the teaching injection video, and learns to obtain the mapping relationship between the syringe posture information and the target posture information to be injected and the action sequence.

[0077] The control module 1004 is used to control the robot arm to execute actions in the action sequence.

[0078] In some embodiments of the present disclosure, an action may include an action type and an action parameter.

[0079] In some embodiments of the present disclosure, the control module 1004 is specifically used to:

[0080] S1, controlling the robot arm to execute the first N actions in the action sequence, and obtaining the injection image after executing the actions; wherein N is a positive integer;

[0081] S2, taking the position information of the syringe in the injection image after the action is performed as new position information of the syringe, taking the position information of the target to be injected in the injection image after the action is performed as new position information of the target to be injected, inputting the new position information of the syringe and the new position information of the target to be injected into the multimodal large model LLaVA to obtain a new action sequence;

[0082] S3, controlling the robot arm to execute the first N actions in the new action sequence;

[0083] S4, repeat steps S2-S3 until the robotic arm performs the preset injection end action.

[0084] In some embodiments of the present disclosure, Fig.10On the basis of the illustrated embodiment, the motion reasoning device for immunization injection may also include a training device. The training device is used to: segment and sample the teaching injection video to obtain continuous video frame images; determine the initial syringe posture information of the syringe and the initial target posture information of the target to be injected in the teaching injection video; the multimodal large model LLaVA uses visual question-answering technology to select the initial action in the continuous video frame image from multiple injection actions in the action skill library through multiple rounds of dialogue tasks, and generates an initial action sequence corresponding to the initial syringe posture information and the initial target posture information to be injected based on the initial action; obtain the action sequence label in the teaching injection video, and the action sequence label is determined based on the robot vaccine injection operation skill knowledge; train the multimodal large model LLaVA based on the action sequence label and the initial action sequence, so that the multimodal large model LLaVA learns to obtain the mapping relationship between the syringe posture information and the target posture information to be injected and the action sequence.

[0085] In some embodiments of the present disclosure, the syringe posture information includes the position information and spatial posture information of the syringe, and the spatial posture information of the syringe is obtained by: establishing a 3D model of the syringe; using a deep learning-based visual algorithm to extract the feature information of the syringe in the current injection image; matching the feature information with the 3D model to capture the spatial posture information of the syringe.

[0086] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0087] In order to implement the above embodiments, the present disclosure also proposes an electronic device, including: a processor, and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the method provided by the above embodiments.

[0088] In order to implement the above embodiments, the present disclosure further proposes a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the methods provided by the above embodiments.

[0089] In order to implement the above embodiments, the present disclosure also proposes a computer program product, including a computer program, which implements the methods provided by the above embodiments when executed by a processor.

[0090] In the description of the aforementioned embodiments, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0091] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present disclosure, "plurality" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0092] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present disclosure includes additional implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present disclosure belong.

[0093] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute the instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0094] It should be understood that the various parts of the present disclosure can be implemented in hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used to implement: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0095] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0096] In addition, each functional unit in each embodiment of the present disclosure may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0097] The storage medium mentioned above may be a read-only memory, a disk or an optical disk, etc. Although the embodiments of the present disclosure have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations of the present disclosure. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present disclosure.

Claims

1. A motion reasoning method for immune injection, the method is applied to an injection robot, the injection robot controls the action of the syringe through a mechanical arm, characterized in that: The following steps are involved: Acquire a current injection image, wherein the current injection image includes a syringe and a target to be injected; Determine the syringe position information and the target position information to be injected according to the current injection image; The syringe posture information and the target posture information to be injected are input into a pre-trained multimodal large model LLaVA to obtain an action sequence corresponding to the syringe posture information and the target posture information to be injected, wherein the action sequence includes a plurality of actions; the multimodal large model LLaVA uses visual question-answering technology to conduct a multi-round dialogue on the teaching injection video to accumulate content understanding, and learns to obtain a mapping relationship between the syringe posture information, the target posture information to be injected, and the action sequence; Control the robot arm to execute the actions in the action sequence.

2. The method according to claim 1, characterized in that The controlling the robot arm to perform the actions in the action sequence includes: S1, controlling the robot arm to execute the first N actions in the action sequence to obtain an injection image after the actions are executed; wherein N is a positive integer; S2, taking the posture information of the syringe in the injection image after the action is performed as new syringe posture information, taking the posture information of the target to be injected in the injection image after the action is performed as new target posture information to be injected, inputting the new syringe posture information and the new target posture information to be injected into the multimodal large model LLaVA to obtain a new action sequence; S3, controlling the robot arm to execute the first N actions in the new action sequence; S4, repeating steps S2-S3 until the robotic arm performs a preset injection ending action.

3. The method according to claim 1, characterized in that The multimodal large model LLaVA uses visual question-answering technology to understand the content of multiple rounds of dialogue on the teaching injection video, and learns to obtain the mapping relationship between the syringe posture information, the posture information of the target to be injected, and the action sequence, including: Segmenting and sampling the teaching injection video to obtain continuous video frame images; Determine the initial syringe posture information of the syringe and the initial target posture information of the target object to be injected in the teaching injection video; The multimodal large model LLaVA adopts visual question-answering technology, selects the initial action in the continuous video frame image from multiple injection actions in the action skill library through multiple rounds of dialogue tasks, and generates an initial action sequence corresponding to the initial syringe posture information and the initial target posture information to be injected based on the initial action; Obtaining action sequence labels in the teaching injection video, wherein the action sequence labels are determined based on robot vaccine injection operation skill knowledge; The multimodal large model LLaVA is trained based on the action sequence label and the initial action sequence, so that the multimodal large model LLaVA learns to obtain the mapping relationship between the syringe posture information and the target posture information to be injected and the action sequence.

4. The method according to claim 1, characterized in that: The syringe posture information includes the position information and spatial posture information of the syringe, and the spatial posture information of the syringe is obtained by: Establishing a 3D model of the syringe; Using a deep learning-based visual algorithm to extract feature information of the syringe in the current injection image; The feature information is matched with the 3D model to capture the spatial posture information of the syringe.

5. The method according to claim 1, characterized in that The action includes an action type and an action parameter.

6. A motion reasoning device for immunization injection, the device is used to control the motion of the syringe through the mechanical arm of the injection robot, characterized in that: The device comprises: An acquisition module, used for acquiring a current injection image, wherein the current injection image includes a syringe and a target to be injected; A first determination module, used to determine the syringe posture information and the to-be-injected target posture information according to the current injection image; The second determination module is used to input the syringe posture information and the target posture information to be injected into a pre-trained multimodal large model LLaVA to obtain an action sequence corresponding to the syringe posture information and the target posture information to be injected, wherein the action sequence includes a plurality of actions; the multimodal large model LLaVA uses visual question-answering technology to conduct a multi-round dialogue on the teaching injection video to accumulate content understanding, and learn to obtain a mapping relationship between the syringe posture information, the target posture information to be injected, and the action sequence; A control module is used to control the robot arm to execute the actions in the action sequence.

7. The device according to claim 6, characterized in that The control module is specifically used for: S1, controlling the robot arm to execute the first N actions in the action sequence to obtain an injection image after the actions are executed; wherein N is a positive integer; S2, taking the posture information of the syringe in the injection image after the action is performed as new syringe posture information, taking the posture information of the target to be injected in the injection image after the action is performed as new target posture information to be injected, inputting the new syringe posture information and the new target posture information to be injected into the multimodal large model LLaVA to obtain a new action sequence; S3, controlling the robot arm to execute the first N actions in the new action sequence; S4, repeating steps S2-S3 until the robotic arm performs a preset injection ending action.

8. The device according to claim 6, characterized in that The device also includes a training device; the training device is used for: Segmenting and sampling the teaching injection video to obtain continuous video frame images; Determine the initial syringe posture information of the syringe and the initial target posture information of the target object to be injected in the teaching injection video; The multimodal large model LLaVA adopts visual question-answering technology, selects the initial action in the continuous video frame image from multiple injection actions in the action skill library through multiple rounds of dialogue tasks, and generates an initial action sequence corresponding to the initial syringe posture information and the initial target posture information to be injected based on the initial action; Obtaining action sequence labels in the teaching injection video, wherein the action sequence labels are determined based on robot vaccine injection operation skill knowledge; The multimodal large model LLaVA is trained based on the action sequence label and the initial action sequence, so that the multimodal large model LLaVA learns to obtain the mapping relationship between the syringe posture information and the target posture information to be injected and the action sequence.

9. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 5.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 5 when executed by a processor.

Images