Material stacking method, device and equipment based on large model and medium

Abstract

The invention discloses a material stacking method, device and equipment based on a large model and a medium. The method comprises the following steps: converting a voice instruction input by a user into a text instruction through a preset language interaction model; the text instruction and a corresponding target collection image are analyzed and processed through a preset image processing model, and space information of the to-be-stacked materials and state information of a stacking storage area are obtained; the text instruction, the space information of the to-be-stacked materials and the state information of the stacking storage area are used for generating stacking control codes through a preset large model; and controlling a robot to stack the to-be-stacked materials according to the stacking control code. By implementing the method, the problems that an industrial robot in the prior art is high in use difficulty and cannot be applied to complex stacking tasks can be solved.

Description

Technical Field

[0001] The present invention relates to the technical field of material palletizing, and in particular to a material palletizing method, device, equipment and medium based on a large model. Background Art

[0002] With the continuous advancement of robotics technology, industrial robots are increasingly used in various fields, especially in palletizing operations, showing great potential for efficient handling of various objects. However, when performing palletizing tasks, industrial robots in the prior art are highly dependent on control codes manually compiled by users according to specific palletizing requirements. Unfortunately, once the shape, specifications or palletizing standards of the materials change, the user must re-intervene, manually adjust the control code, and go through multiple tedious debugging processes to ensure that the robot can adapt to the new operating requirements. This frequent and complex adjustment requirement for the control code not only increases the difficulty of operation, but also limits the ability of industrial robots to flexibly respond to diverse and complex palletizing tasks, making it difficult for existing industrial robots to adapt to complex and diverse palletizing tasks. Summary of the invention

[0003] The embodiments of the present invention provide a large-model-based material stacking method, device, equipment and medium, aiming to solve the problem that industrial robots in the prior art are difficult to use and cannot be applied to complex stacking tasks.

[0004] In the first aspect, an embodiment of the present invention provides a material stacking method based on a large model, which includes: converting a voice command input by a user into a text command through a preset language interaction model; analyzing and processing the text command and the corresponding target acquisition image through a preset image processing model to obtain spatial information of the material to be stacked and status information of the stacking storage area; generating a stacking control code from the text command, the spatial information of the material to be stacked, and the status information of the stacking storage area through a preset large model; and controlling a robot to stack the material to be stacked according to the stacking control code.

[0005] In the second aspect, an embodiment of the present invention further provides a material stacking device based on a large model, which includes: a conversion unit, used to convert the voice instructions input by the user into text instructions through a preset language interaction model; an analysis unit, used to analyze and process the text instructions and the corresponding target acquisition image through a preset image processing model to obtain the spatial information of the material to be stacked and the status information of the stacking storage area; a generation unit, used to generate a stacking control code through a preset large model based on the text instructions, the spatial information of the material to be stacked and the status information of the stacking storage area; a stacking unit, used to control the robot to stack the material to be stacked according to the stacking control code.

[0006] In a third aspect, an embodiment of the present invention further provides a computer device, which includes a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the above method when executing the computer program.

[0007] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the storage medium stores a computer program, wherein the computer program includes program instructions, and the program instructions can implement the above method when executed by a processor.

[0008] The embodiment of the present invention provides a material stacking method, device, equipment and medium based on a large model. The method includes: converting the voice command input by the user into a text command through a preset language interaction model; analyzing and processing the text command and the corresponding target acquisition image through a preset image processing model to obtain the spatial information of the material to be stacked and the status information of the stacking storage area; generating a stacking control code through a preset large model for the text command, the spatial information of the material to be stacked and the status information of the stacking storage area; and controlling the robot to stack the material to be stacked according to the stacking control code. The embodiment of the present invention converts the user's voice command into a text command, and analyzes and processes the text command and the acquisition image through an image processing model to obtain the information of the material to be stacked and the stacking storage area, and generates a stacking control code through a preset large model for the obtained information, thereby avoiding the user's writing of more complex codes and the process of teaching the industrial robot, thereby reducing the difficulty of using the industrial robot, so that the robot can easily face a variety of complex stacking scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0010] Figure 1 A schematic diagram of a process flow of a material stacking method based on a large model provided in an embodiment of the present invention;

[0011] Figure 2 A schematic diagram of a sub-process of a material stacking method based on a large model provided in an embodiment of the present invention;

[0012] Figure 3 A schematic diagram of a sub-process of a material stacking method based on a large model provided in an embodiment of the present invention;

[0013] Figure 4 A schematic diagram of a sub-process of a material stacking method based on a large model provided in an embodiment of the present invention;

[0014] Figure 5 A schematic diagram of a sub-process of a material stacking method based on a large model provided in an embodiment of the present invention;

[0015] Figure 6 A schematic diagram of the positions of stacking points of a material stacking method based on a large model provided in an embodiment of the present invention;

[0016] Figure 7 A schematic diagram of a sub-process of a material stacking method based on a large model provided in an embodiment of the present invention;

[0017] Figure 8 A schematic diagram of a sub-process of a material stacking method based on a large model provided in an embodiment of the present invention;

[0018] Fig. 9 A schematic block diagram of a material stacking device based on a large model provided in an embodiment of the present invention;

[0019] Fig.10 A schematic block diagram of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0020] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0022] It should also be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0023] It should be further understood that the term "and / or" used in the present description and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0024] See also Figure 1 , Figure 1 A schematic flow chart of a large-model-based material palletizing method provided in an embodiment of the present invention. The large-model-based material palletizing method in this embodiment can be applied to the control of a palletizing robot, wherein the palletizing task is acquired through the language interaction model of the palletizing robot, and the code for controlling the robot is generated according to the acquired palletizing task through the image processing model and the preset large model, so that the user does not need to write more complex codes, and avoids the industrial robot teaching process, thereby reducing the difficulty of using the industrial robot.

[0025] Figure 1 1 is a schematic flow chart of a material stacking method based on a large model provided by an embodiment of the present invention. As shown in the figure, the method includes the following steps S110-S140.

[0026] S110: Convert the voice command input by the user into a text command through a preset language interaction model.

[0027] In this embodiment, the preset language interaction model refers to a model for realizing interaction between humans and devices through natural language. The preset language interaction model is an artificial intelligence system, including a speech recognition model, a speech synthesis model, etc. The user's voice input is converted into text through the speech recognition model, and the response text can be converted into a natural and fluent speech output through the speech synthesis model. Among them, the specific training process of the speech recognition model and the speech synthesis model is not limited, and the corresponding effect can be achieved. The user sends a voice instruction with a palletizing task to the palletizing robot through natural language, and the natural language is converted into a text instruction through the speech recognition model in the preset language interaction model. Among them, the voice instruction needs to describe the characteristic description of the material to be palletized and the description of the requirements for palletizing. For example, the voice instruction input by the user is "Put the express boxes in 3 rows and 3 columns in 2 layers and stack them on the wooden board", then the speech recognition model can convert it into a text instruction. By converting the voice instruction input by the user into a text instruction, a specific and understandable execution instruction is obtained, and a corresponding control code is generated according to the execution instruction.

[0028] S120, analyzing and processing the text instruction and the corresponding target acquisition image through a preset image processing model to obtain spatial information of the materials to be palletized and status information of the palletizing storage area.

[0029] In this embodiment, the target acquisition image is an image of the material to be palletized and the image of the palletizing storage area collected according to the text instruction. Specifically, the text instruction carries the material to be palletized and the area where it will be stored. After obtaining the text instruction, the robot will automatically collect image data of the material to be palletized and the palletizing storage area through a visual sensor, and use it as the target acquisition image. The preset image processing model refers to a computing model for processing and analyzing images. Specifically, the preset image processing model in this embodiment can be a VLM (Vision-Language Models) multimodal large model, or a neural network model, wherein the VLM multimodal large model is an artificial intelligence model that can process image and text data simultaneously. This model integrates information from both visual and language modalities and can perform complex cross-modal tasks. Specifically, the VLM multimodal large model can be trained in advance through training images so that it can determine information such as the location, size, and quantity of the target, wherein the specific training process is not limited. The text instruction and the corresponding target acquisition image are input as input to the preset image processing model for analysis and processing, so as to obtain the spatial information of the materials to be palletized, such as the position and size of the materials to be palletized, and the status information of the palletizing storage area, such as the position of the palletizing storage and the number of palletized materials. The preset image processing model is used for analysis and processing to obtain the spatial information of the materials to be palletized and the status information of the palletizing storage area, providing a data basis for the subsequent generation of the robot control code.

[0030] In one embodiment, if Figure 2 As shown, the step S120 also includes steps S1201-S1203 before the step S120.

[0031] S1201, searching for a corresponding material to be palletized among a plurality of materials according to the characteristic information of the material to be palletized, and performing image acquisition on the material to be palletized to obtain an image of the material to be palletized;

[0032] S1202, searching for a corresponding palletizing storage area according to the palletizing requirement information, and performing image acquisition on the palletizing storage area to obtain an image of the palletizing storage area;

[0033] S1203: Determine the target acquisition image according to the image of the material to be palletized and the image of the palletizing storage area.

[0034] In this embodiment, the text instruction includes characteristic information of materials to be palletized and the palletizing requirement information. The characteristic information of materials to be palletized defines the attribute information of materials, such as the category and quantity to be palletized. For example, the characteristic information of materials to be palletized may be 18 express boxes. The palletizing requirement information is information defining the palletizing method and the palletizing position of materials, such as palletizing the materials in a 3×3×2 (3 rows, 3 columns, 2 layers) manner to storage area A. According to the characteristic information of materials to be palletized, the corresponding materials to be palletized are searched among a number of materials stored according to the information of the attributes of the materials to be palletized, and the image information of the materials to be palletized is collected through a visual sensor; and the corresponding palletizing storage area is found according to the palletizing position in the palletizing requirement, and the image of the palletizing storage area is collected to obtain the image of the palletizing storage area, and the target acquisition image is determined based on the image of the materials to be palletized and the image of the palletizing storage area. By collecting image data according to the information corresponding to the text instruction, the specific size information of the material and the spatial information of the palletizing storage area are obtained.

[0035] In one embodiment, if Figure 3 As shown, the step S120 also includes steps S121-S122.

[0036] S121, analyzing and processing the characteristic information of the material to be stacked and the image of the material to be stacked through the preset image processing model to obtain the position information and size information of the material to be stacked;

[0037] S122, analyzing and processing the palletizing requirement information and the palletizing storage area image through the preset image processing model to obtain the position information of the palletizing storage area and the quantity information of the palletized materials.

[0038] In this embodiment, the characteristic information of the material to be palletized and the image of the material to be palletized are both input to the preset image processing model at the same time. After they are input into the model together, the model can generate the position information and size information of the material to be palletized and output it. Specifically, the preset image processing model can preprocess the input image of the material to be palletized, including image enhancement, denoising, grayscale and other preprocessing operations to obtain a high-quality image, and extract features from the preprocessed image through a feature extraction module to obtain feature information related to the material to be palletized, such as the edge, contour, color, texture and other information of the material, and determine whether it matches the characteristic information of the material to be palletized based on the feature information. If it matches, the position detection algorithm in the image processing model, such as edge detection, contour tracking, etc., is used to determine the position of the material in the image coordinate system, and the position in the image coordinate system is converted into the position in the base coordinate system of the robot based on the calibration parameters of the camera and the mapping relationship between the image coordinate system and the base coordinate system of the robot. And by using the dimension measurement algorithm in the image processing model, according to the extracted material feature information, the physical dimensions of the material such as length, width, height, etc. are calculated, thereby obtaining the position information and size information of the material to be stacked. Among them, the specific analysis and processing flow of the preset image processing model is not limited, and it is sufficient to obtain accurate position information and size information of the material to be stacked. The stacking requirement information and the image of the stacking storage area are used as the same input and input into the preset image processing model for analysis and processing. Among them, the specific analysis and processing flow is the same as the above-mentioned flow for obtaining the position information and size information of the material to be stacked, and will not be repeated here. It should be noted that the position of the stacking storage area is only a marking point, not an area. The image of the stacking storage area is obtained by using a visual sensor. After analysis and processing by the preset image processing model, the coordinates in the coordinate system established with the base of the robot (usually the mounting base of the robot) as the origin are obtained. The subsequent robot can directly use this coordinate to move to the position. The preset image processing model is used to analyze and process the location information and size information of the materials to be palletized, as well as the location information of the palletizing storage area and the quantity information of the palletized materials, so as to generate the corresponding control code to control the robot for palletizing.

[0039] S130, generating a palletizing control code by using a preset large model according to the text instruction, the space information of the material to be palletized and the status information of the palletizing storage area.

[0040] In this embodiment, the preset large model is a large-scale language model based on deep learning technology, which has strong learning and reasoning capabilities. In this embodiment, the preset large model is a basic large model. The basic large model can automatically learn and process a large amount of natural language text data through deep learning technology to generate natural and fluent language text. The text instructions, the spatial information of the materials to be stacked and the status information of the stacking storage area are used to generate stacking control codes through the preset large model. Specifically, after the large model receives the text instructions, the spatial information of the materials to be stacked and the status information of the stacking storage area, it will pre-process and feature extract the input information to capture the key features such as the position, size, stacking requirements and the status of the storage area of ​​the material. Then, the model will apply a series of optimization algorithms, such as path planning algorithms, collision detection algorithms, etc., to determine the best stacking strategy and the motion trajectory of the robot. These algorithms will comprehensively consider multiple factors, such as the efficiency of material handling, the stability of stacking, the kinematic constraints of the robot, etc., to generate stacking control codes. The control code consists of a series of basic action API functions, each of which corresponds to a specific action performed by the robot, such as grabbing materials, moving to a specified location, placing materials, etc. These basic action API functions are combined in a predetermined order and parameters to form a complete palletizing control code. The control code is generated by presetting a large model based on the text instructions, the spatial information of the materials to be palletized, and the status information of the palletizing storage area to ensure that the generated palletizing control code is both efficient and accurate, thereby improving the efficiency and accuracy of the palletizing operation.

[0041] In one embodiment, if Figure 4 As shown, the step S130 also includes steps S131-S132.

[0042] S131, constructing a prompt instruction for the preset large model according to the text instruction, the spatial information of the materials to be stacked, the state information of the stacking storage area, and the preset basic function, the preset output requirement, and the preset sample;

[0043] S132, inputting the prompt instruction into the preset large model to obtain a palletizing control code that meets the preset output requirement.

[0044] In this embodiment, the prompt instruction is an input text or instruction provided to the model to guide the model to generate a specific type of response. Specifically, the large model only accepts one type of input, namely the prompt instruction (Prompt). All engineering work related to the large model is carried out around the prompt instruction. Therefore, it is necessary to construct the prompt instruction of the preset large model according to the text instruction, the spatial information of the material to be stacked, the status information of the stacking storage area, and the preset basic function, the preset output requirement, and the preset sample. Specifically, the prompt instruction consists of three parts: task description, output format, and sample example. Among them, the task description defines the role of the task as an industrial robot stacking code generation assistant, gives the task instructions as output functions and replies, and explains the function information related to the task. The output format describes the output requirements in what form the output is output, and specifies the operation keys and execution keys of the output requirements. The sample example gives standard replies for input and output, which can be used for the large model to learn and imitate. Specifically, the voice input by the user is converted into text and used as the task description, the preset basic function and the preset output requirement are used as the output format, the preset sample is used as the sample example, the spatial information of the material to be palletized and the state information of the palletizing storage area are used as the context, and other background information related to the task, thereby jointly constituting the prompt instruction of the preset large model, and the prompt instruction is input into the preset large model, and the large model will generate the palletizing control code that meets the preset output requirements according to the preset sample generation and the preset basic function, for example:

[0045] #Task Description

[0046] instruction="…"

[0047] You are an industrial robot palletizing code generation assistant, and the robot has some built-in functions. Your task is to output the functions to be run and the responses in json format according to my instructions.

[0048] Introduction to built-in functions:

[0049] Palletizing point generation function: point_generation()

[0050] Stacking execution function: stacking() ...

[0052] #Output format

[0053] output format=…

[0054] In the 'function' key, a list of function names is output, representing the function name and parameters to be run

[0055] In the 'response' key, according to my instructions, output your reply to me in the first person

[0056] # Few-sample examples

[0057] examples="…"

[0058] My instructions: Stack the gifts into boxes in a 2x2x3 pattern.

[0059] You output in this format:

[0060] {

[0061] 'function':[point_generation(2,2,3,l 礼品 ,w 礼品 ,h 礼品 ,x 箱子 ,y 箱子 ,z 箱子 )','stacking(x 礼品 ,y 礼品 ,z 礼品 ,A,0)']

[0062] response: Received the instruction, the gifts are being stacked into the box, please keep a safe distance

[0063] }

[0064] Among them, 2,2,3 are the palletizing methods, l 礼品 ,w 礼品 ,h 礼品 For gift size information, x 箱子 ,y 箱子 ,z 箱子 is the location information of the box, x 礼品 ,y 礼品 ,z 礼品 is the location information of the gift, A,0 is the location and quantity of the gift, where the above code is only an example of prompt instruction, not a specific limitation, so no excessive explanation is given for the parameters. By generating the stacking control code through the large model, the user does not need to write more complex code, and avoids the teaching process of the industrial robot, thereby reducing the difficulty of using the industrial robot.

[0065] In one embodiment, if Figure 5 As shown, the step S132 also includes steps S1321-S1324.

[0066] S1321, using the size information of the material to be palletized, the palletizing requirement information, and the position information of the palletizing storage area as receiving parameters of the palletizing point generation function;

[0067] S1322, obtaining a palletizing point generation code through the palletizing point generation function according to the received parameters, wherein the palletizing point generation code generates a palletizing point for each of the materials to be palletized to be stored in the palletizing storage area;

[0068] S1323, generating a palletizing execution code through the palletizing execution function according to the position information of the material to be palletized, the palletizing point, and the quantity information of the palletized material;

[0069] S1324. Determine the palletizing control code according to the palletizing point generation code and the palletizing execution code.

[0070] In this embodiment, the preset basic function includes a palletizing point generation function and a palletizing execution function, wherein the palletizing point generation function is used to generate the location point (palletizing point) of the material to be palletized in the palletizing storage area, and the palletizing execution function is used to control how the robot performs palletizing. Specifically, the palletizing point generation function point_generation(L,W,H,l 物料 ,w 物料 ,h 物料 ,x 存放 ,y 存放 ,z 存放 ) is a function that generates the three-dimensional coordinates of the stacking point. The received parameters are the stacking method (L (row) × W (column) × H (layer)), the size information of the material to be stacked (l 物料 ,w 物料 ,h 物料 ), the location information of the palletizing storage area (x 存放 ,y 存放 ,z 存放 ). The palletizing method is used to define the matrix A that stores the three-dimensional coordinates of the palletizing points. The size of the matrix A is L×W×H. The palletizing points in the palletizing storage area are used as the values ​​of the matrix A[0][0][0]. The size of the material to be palletized is l 物料 ,w 物料 ,h 物料They are respectively used as the increment values ​​along the positive direction of the X-axis, Y-axis and Z-axis. The size of the material to be palletized is taken as the increment along the positive direction of the X-axis, Y-axis and Z-axis. For each palletizing point in the matrix A, it has its coordinates. Specifically, the axis coordinates of the i-th row: xi=x+i*l; the axis coordinates of the j-th column: yj=y+j*w; the axis coordinates of the k-th layer: zk=z+k*h. The value range of i is from 0 to L-1, the value range of j is from 0 to W-1, and the value range of k is from 0 to H-1. For example, the palletizing method is to place each layer in a 3*3 manner, and two layers are placed. The size information of the material to be palletized is (250, 250, 100), and the location of the palletizing storage area is (150, 150, 0). Then the function generates the position of each palletizing point, such as Figure 6 The stacking point position diagram is shown in FIG. Among them, the stacking generation function is a preset function, which is not limited to this function. It can generate the corresponding code according to the above-mentioned receiving parameters and methods. The stacking execution function stacking(x 物料 ,y 物料 ,z 物料 ,A,n) is an encapsulated function, which is used to realize the process of grabbing the materials to be stacked and placing them at various stacking points. The receiving parameters of the stacking execution function are the location information of the materials to be stacked, the stacking point matrix A output by the stacking point generation function, and the number of materials already in the storage area. The stacking execution code is generated through the stacking execution function according to the location information of the materials to be stacked, the stacking points, and the number of materials already stacked. For example, there are 18 materials to be stacked, and each layer is arranged in a 3*3 manner, and two layers are arranged. If there are already 9 stacked materials, then skip the positions A[0][0[0] to A[2][2][0] (that is, all the stacking points on the first layer) and generate stacking(x 物料 ,y 物料 ,z 物料 ,A,10), where A is A[0][0[1] (the first palletizing point on the second layer). The palletizing control code is determined based on the palletizing point generation code and the palletizing execution code. By generating the palletizing control code, the robot can palletize materials according to the palletizing store and the palletizing storage location.

[0071] S140: Controlling the robot to palletize the materials to be palletized according to the palletizing control code.

[0072] In this embodiment, the robot moves according to the instructions of the generated palletizing control code to grab the materials to be palletized, and moves to the designated palletizing point according to the trajectory planned by the palletizing control code. After arriving at the palletizing point, the robot will place the materials at a predetermined position according to the instructions of the palletizing control code to complete the palletizing of the materials. After completing the material grabbing, moving and placing, the robot will return to the material storage area and repeat the above steps until all the materials to be palletized are completed. By controlling the robot to palletize the materials to be palletized according to the palletizing control code, the robot can perform accurate palletizing through a reasonable palletizing strategy.

[0073] In one embodiment, if Figure 7 As shown, the step S140 includes steps S141-S142.

[0074] S141, controlling the robot to obtain the material to be stacked according to the position information of the material to be stacked;

[0075] S142: Eliminate the occupied stacking points according to the quantity information of the stacked materials, and move the materials to be stacked to unoccupied stacking points.

[0076] In this embodiment, the robot transports the material to be palletized through the palletizing execution function in the palletizing control code. Specifically, the robot moves to a safety point above the material to be palletized according to the position information of the material to be palletized, and descends to the grabbing point of the material to be palletized to grab the material to be palletized, and then moves to the safety point above the material to be palletized again to transport the material to be palletized, wherein the safety point and the grabbing point can be set according to the specific material to be palletized, and there is no limitation on this. The materials are placed according to the matrix A storing the three-dimensional coordinates of the stacking points and the quantity information of the stacked materials. Specifically, the occupied stacking points in the matrix A are removed. If there are 18 materials to be stacked, each layer is arranged in the manner of 3 (rows) × 3 (columns) × 2 (layers), and there are 9 stacked materials, then the first layer is already full. At this time, the positions A[0][0[0], A[1][0[0], A[][1[0], A[1][1[0] to A[2][2][0] of the first layer in the matrix A (i.e., all the stacking points on the first layer) are occupied points. The occupied stacking points are removed, and the robot is controlled to directly place the material to be stacked at the stacking point A[0][0][1] (the first position on the second layer) on the second layer, and the steps of moving to a safe point above the material to be stacked are repeated to carry out material handling. By controlling the robot to move the materials to be stacked to unoccupied stacking points, there is no need to place the materials to be stacked in a fixed position input by humans, which is conducive to the development and promotion of flexible production lines.

[0077] In one embodiment, if Figure 8 As shown, the step S140 also includes steps S1401-S1402.

[0078] S1401, judging whether palletizing abnormality occurs according to the sensor detection information;

[0079] S1402: If an abnormality occurs, a control signal of the robot is triggered to control the robot to report the abnormality.

[0080] In this embodiment, the sensor detection information is information detected by the sensor pre-installed on the robot. Whether a stacking abnormality occurs is determined based on the sensor detection information. For example, the material position sensor can be used to detect whether the material is in a predetermined position. If the material to be stacked is not found in the predetermined position, it is determined that a stacking abnormality occurs. The stacking abnormality is an abnormal situation that occurs when the robot performs stacking, including but not limited to abnormal shutdown, lack of incoming materials, and the end of stacking. If an abnormality occurs, the control signal of the robot is triggered. When the control signal is triggered, the robot will be instructed to broadcast the abnormality. Specifically, the abnormal situation can be broadcasted through the speech synthesis model in the preset language conversion model in step S110. By determining whether an abnormality occurs and controlling the robot to broadcast an abnormality when an abnormality occurs, the user is reminded of the abnormal stacking state of the current robot in a timely manner, so as to facilitate timely adjustment of the robot or the stacking task.

[0081] Fig. 9 is a schematic block diagram of a material stacking device 200 based on a large model provided by an embodiment of the present invention. Fig. 9 As shown, corresponding to the above-mentioned material stacking method based on a large model, the present invention also provides a material stacking device based on a large model. The material stacking device based on a large model includes a unit for executing the above-mentioned material stacking method based on a large model, and the device can be configured in a terminal such as a desktop computer, a tablet computer, a laptop computer, etc. For details, please refer to Fig. 9 The material palletizing device based on the large model includes a conversion unit 210, an analysis unit 220, a generation unit 230 and a palletizing unit 240.

[0082] The conversion unit 210 is used to convert the voice command input by the user into a text command through a preset language interaction model.

[0083] The analysis unit 220 is used to analyze and process the text instruction and the corresponding target acquisition image through a preset image processing model to obtain the spatial information of the materials to be palletized and the status information of the palletizing storage area.

[0084] In one embodiment, the analyzing unit 220 includes a searching unit, a finding unit, and a determining unit.

[0085] A search unit, used for searching corresponding materials to be stacked among a plurality of materials according to the characteristic information of the materials to be stacked, and performing image acquisition on the materials to be stacked to obtain images of the materials to be stacked;

[0086] A searching unit, used for searching a corresponding palletizing storage area according to the palletizing requirement information, and performing image acquisition on the palletizing storage area to obtain an image of the palletizing storage area;

[0087] A determination unit is used to determine the target acquisition image according to the image of the material to be palletized and the image of the palletizing storage area.

[0088] In one embodiment, the analysis unit 220 includes a first processing unit and a second processing unit.

[0089] A first processing unit is used to analyze and process the characteristic information of the material to be stacked and the image of the material to be stacked through the preset image processing model to obtain the position information and size information of the material to be stacked;

[0090] The second processing unit is used to analyze and process the palletizing requirement information and the palletizing storage area image through the preset image processing model to obtain the position information of the palletizing storage area and the quantity information of the palletized materials.

[0091] The generating unit 230 is used to generate a palletizing control code by using a preset large model according to the text instruction, the space information of the material to be palletized and the status information of the palletizing storage area.

[0092] In one embodiment, the generating unit 230 includes a constructing unit and an acquiring unit.

[0093] A construction unit, used for constructing prompt instructions of the preset large model according to the text instructions, the spatial information of the materials to be stacked, the state information of the stacking storage area, and the preset basic function, the preset output requirements, and the preset samples;

[0094] The acquisition unit is used to input the prompt instruction into the preset large model to obtain the palletizing control code that meets the preset output requirements.

[0095] In one embodiment, the generating unit 230 includes a receiving unit, a first generating unit, a second generating unit, and a determining unit.

[0096] A receiving unit, used for taking the size information of the material to be palletized, the palletizing requirement information and the position information of the palletizing storage area as receiving parameters of the palletizing point generation function;

[0097] A first generating unit, configured to obtain a palletizing point generating code through the palletizing point generating function according to the received parameters, wherein the palletizing point generating code generates a palletizing point for each of the materials to be palletized to be stored in the palletizing storage area;

[0098] A second generating unit is used to generate a palletizing execution code through the palletizing execution function according to the position information of the material to be palletized, the palletizing point and the quantity information of the palletized material;

[0099] A determination unit is used to determine the palletizing control code according to the palletizing point generation code and the palletizing execution code.

[0100] The palletizing unit 240 is used to control the robot to palletize the materials to be palletized according to the palletizing control code.

[0101] In one embodiment, the palletizing unit 240 includes a rejecting unit and a moving unit.

[0102] A rejection unit, used for controlling the robot to obtain the material to be stacked according to the position information of the material to be stacked;

[0103] The moving unit is used to remove the occupied stacking points according to the quantity information of the stacked materials, and move the materials to be stacked to unoccupied stacking points.

[0104] In one embodiment, the palletizing unit 240 further includes a judging unit and a reporting unit.

[0105] A judgment unit, used to judge whether palletizing abnormality occurs according to the sensor detection information;

[0106] The reporting unit is used to trigger the control signal of the robot if an abnormality occurs, and control the robot to report the abnormality.

[0107] It should be noted that technicians in the relevant field can clearly understand that the specific implementation process of the above-mentioned large-model-based material stacking device 200 and each unit can refer to the corresponding description in the aforementioned method embodiment, and for the convenience and conciseness of the description, it will not be repeated here.

[0108] The above-mentioned material stacking device based on the large model can be implemented in the form of a computer program, which can be used in Fig.10 Runs on the computer device shown.

[0109] See also Fig.10 , Fig.105 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device 500 may be a terminal or a server, wherein the terminal may be an electronic device with communication functions such as a smart phone, a tablet computer, a laptop computer, a desktop computer, a personal digital assistant, and a wearable device. The server may be an independent server or a server cluster composed of multiple servers.

[0110] See also Fig.10 The computer device 500 includes a processor 502 , a memory and a network interface 505 connected via a system bus 501 , wherein the memory may include a non-volatile storage medium 503 and an internal memory 504 .

[0111] The non-volatile storage medium 503 can store an operating system 5031 and a computer program 5032. The computer program 5032 includes program instructions, and when the program instructions are executed, the processor 502 can execute a material palletizing method based on a large model.

[0112] The processor 502 is used to provide computing and control capabilities to support the operation of the entire computer device 500 .

[0113] The internal memory 504 provides an environment for the operation of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute a material stacking method based on a large model.

[0114] The network interface 505 is used to communicate with other devices over the network. Fig.10 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present application, and does not constitute a limitation on the computer device 500 to which the solution of the present application is applied. The specific computer device 500 may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0115] The processor 502 is used to run a computer program 5032 stored in the memory to implement the steps of the above method.

[0116] It should be understood that in the embodiment of the present application, the processor 502 may be a central processing unit (CPU), and the processor 502 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0117] It can be understood by those skilled in the art that all or part of the processes in the method for implementing the above embodiment can be completed by instructing the relevant hardware through a computer program. The computer program includes program instructions, and the computer program can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiment of the above method.

[0118] Therefore, the present invention also provides a storage medium. The storage medium may be a computer-readable storage medium. The storage medium stores a computer program, wherein the computer program includes program instructions. When the program instructions are executed by a processor, the processor executes the steps of the above method.

[0119] The storage medium may be a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk, or an optical disk, etc., which are computer-readable storage media that can store program codes.

[0120] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0121] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of each unit is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0122] The steps in the method of the embodiment of the present invention can be adjusted in order, combined and deleted according to actual needs. The units in the device of the embodiment of the present invention can be combined, divided and deleted according to actual needs. In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0123] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, terminal, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention.

[0124] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A material stacking method based on a large model, characterized in that: include: Convert the voice commands input by the user into text commands through a preset language interaction model; The text instructions and the corresponding target acquisition images are analyzed and processed by a preset image processing model to obtain the spatial information of the materials to be stacked and the status information of the stacking storage area; Generate a palletizing control code by using the text instruction, the space information of the material to be palletized and the status information of the palletizing storage area through a preset large model; The robot is controlled according to the palletizing control code to palletize the materials to be palletized.

2. The method according to claim 1, characterized in that The text instruction includes characteristic information of the material to be palletized and palletizing requirement information. Before the step of analyzing and processing the text instruction and the corresponding target acquisition image through a preset image processing model, the method includes: Searching for a corresponding material to be stacked among a plurality of materials according to the characteristic information of the material to be stacked, and performing image acquisition on the material to be stacked to obtain an image of the material to be stacked; Finding a corresponding palletizing storage area according to the palletizing requirement information, and performing image acquisition on the palletizing storage area to obtain an image of the palletizing storage area; The target acquisition image is determined according to the image of the material to be palletized and the image of the palletizing storage area.

3. The method according to claim 2, characterized in that The step of analyzing and processing the text instruction and the corresponding target acquisition image through a preset image processing model to obtain the spatial information of the material to be palletized and the status information of the palletizing storage area includes: Analyze and process the characteristic information of the material to be stacked and the image of the material to be stacked through the preset image processing model to obtain the position information and size information of the material to be stacked; The palletizing requirement information and the palletizing storage area image are analyzed and processed by the preset image processing model to obtain the position information of the palletizing storage area and the quantity information of the palletized materials.

4. The method according to claim 3, characterized in that: The step of generating a palletizing control code by using the text instruction, the spatial information of the material to be palletized and the status information of the palletizing storage area through a preset large model comprises: Constructing a prompt instruction for the preset large model according to the text instruction, the spatial information of the materials to be stacked, the status information of the stacking storage area, and the preset basic function, the preset output requirement, and the preset sample; The prompt instruction is input into the preset large model to obtain a palletizing control code that meets the preset output requirements.

5. The method according to claim 4, characterized in that The preset basic function includes a palletizing point generation function and a palletizing execution function. The step of inputting the prompt instruction into the preset large model to obtain a palletizing control code that meets the preset output requirements includes: Using the size information of the material to be palletized, the palletizing requirement information and the position information of the palletizing storage area as receiving parameters of the palletizing point generation function; Obtaining a palletizing point generation code through the palletizing point generation function according to the received parameters, wherein the palletizing point generation code generates a palletizing point for each of the materials to be palletized to be stored in the palletizing storage area; Generate a palletizing execution code through the palletizing execution function according to the position information of the material to be palletized, the palletizing point and the quantity information of the palletized material; The palletizing control code is determined according to the palletizing point generation code and the palletizing execution code.

6. The method according to claim 5, characterized in that The step of controlling the robot to palletize the materials to be palletized according to the palletizing control code comprises: Controlling the robot to obtain the material to be stacked according to the position information of the material to be stacked; The occupied stacking points are eliminated according to the quantity information of the stacked materials, and the materials to be stacked are moved to unoccupied stacking points.

7. The method according to claim 1, characterized in that After the step of controlling the robot to palletize the materials to be palletized according to the palletizing control code, the method further comprises: Determine whether palletizing abnormality occurs based on sensor detection information; If an abnormality occurs, the control signal of the robot is triggered to control the robot to report the abnormality.

8. A material stacking device based on a large model, characterized in that: include: A conversion unit, used to convert a voice instruction input by a user into a text instruction through a preset language interaction model; An analysis unit, used for analyzing and processing the text instruction and the corresponding target acquisition image through a preset image processing model to obtain spatial information of the materials to be palletized and status information of the palletizing storage area; A generating unit, used for generating a palletizing control code by using a preset large model from the text instruction, the spatial information of the material to be palletized and the status information of the palletizing storage area; The palletizing unit is used to control the robot to palletize the materials to be palletized according to the palletizing control code.

9. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the method according to any one of claims 1 to 7 can be implemented.

Images