A robot navigation and control method and system based on multimodal large model and SLAM collaboration

Abstract

This invention relates to the field of industrial vision robot navigation and control technology, and discloses an industrial vision robot navigation and control method and system based on multimodal large model and SLAM collaboration. The method includes: acquiring map object data, quality work order data, and production map data recognized by the robot's vision, and generating a robot navigation task record; converting map objects into semantic map object units, and determining the retention rate of objects in the current quality re-inspection task based on the quality work order and production map; generating a SLAM navigation control layer based on the retention rate, adjusting the inspection priority of points to be inspected, and generating a navigation execution strategy from the multimodal large model; the SLAM module controls the robot's movement, re-inspection, and detection accordingly, and sends the detection results back to update subsequent navigation processes. This invention can improve the accuracy of temporary object retention, route adjustment, and inspection sequence control in scenarios where flexible production lines coexist with changeover and quality re-inspection.

Description

Technical Field

[0001] This invention relates to the field of industrial vision robot navigation and control technology, and in particular to an industrial vision robot navigation and control method and system based on multimodal large model and SLAM collaboration. Background Technology

[0002] Industrial vision robots are widely used in modern manufacturing environments for tasks such as equipment inspection, product re-inspection, tooling status confirmation, defect identification, and abnormal area review. Existing industrial vision robots typically rely on SLAM maps for navigation and control, and use visual inspection modules to identify objects or workstation status. In relatively stable production line environments, robots can move according to a pre-set inspection point sequence and perform tasks such as taking photos, inspections, or status acquisition upon reaching the corresponding location. However, in flexible manufacturing production lines, the environment is not always stable. Especially in scenarios involving multi-variety, small-batch production or frequent model changes on the same production line, temporary material carts, rework pallets, temporary barriers, changeover fixtures, temporary inspection tables, and tooling storage racks will repeatedly appear on the work surface. Some of these objects only occupy passageways temporarily, while others are directly related to the current quality re-inspection task. For example, if a batch of products has a crimping burr defect, a rework pallet may be temporarily placed near the crimping station; a changeover fixture may remain next to the inspection table for re-inspection; and a barrier may indicate that a certain area has not yet undergone quality processing. In this situation, the robot cannot simply write all temporary objects to the map for a long time, nor can it clear them all at fixed intervals.

[0003] In existing technologies, common processing methods include updating the map based on the reappearance of objects, or setting fixed retention times according to object categories. The former is prone to prematurely releasing areas when objects are temporarily invisible, while the latter is prone to retaining ordinary temporary objects in the map for extended periods. These methods typically only focus on the existence of objects, whether they occupy channels, and the time of their appearance, failing to further determine whether the object is related to quality work orders, defect batches, processes awaiting re-inspection, or related processes in the production map. Furthermore, existing work order systems are usually only used to determine the inspection points the robot needs to reach, and map objects are only used for path avoidance, lacking effective data correlation between the two. When the robot discovers an anomaly in a certain area during quality re-inspection, traditional methods struggle to prevent this detection result from continuing to affect the subsequent map retention status and inspection point order. This can lead to problems such as unreasonable detours, unreasonable re-inspection sequences, unreasonable temporary object removal, or insufficient re-inspection of critical areas in the current quality re-inspection task.

[0004] Therefore, improving the accuracy of temporary object retention, route adjustment, and inspection sequence control for industrial vision robots in flexible production line quality re-inspection scenarios is a technical problem that urgently needs to be solved. Summary of the Invention

[0005] This invention provides a navigation control method and system for industrial vision robots based on multimodal large model and SLAM collaboration. It aims to at least solve the problem that industrial vision robots, when operating on flexible manufacturing lines and undergoing quality re-inspection, cannot determine the true impact of temporary objects or tooling changes on subsequent inspection navigation based on quality work orders and production maps. This results in temporary objects in the semantic map being prematurely removed or mistakenly retained for a long time, causing the robot's navigation route, re-inspection sequence, and point-to-point control effect to fail to meet current quality handling requirements.

[0006] To achieve the above objectives, this invention provides a navigation and control method for industrial vision robots based on multimodal large model and SLAM collaboration, the method comprising the following steps: Acquire key task data of the industrial vision robot in the current quality re-inspection task and establish a robot navigation task record; wherein, the key task data includes map object data, quality work order data and production map data collected by the industrial vision robot; For the robot navigation task record, the map object data is converted into semantic map object units, and the quality impact data and process association data corresponding to each semantic map object unit are determined based on the quality work order data and the production map data. Based on the quality impact data and the process association data, determine the map retention status of each semantic map object unit in the current quality re-inspection task; A SLAM navigation control layer is generated based on the map retention state, and the inspection priority of the points to be inspected is adjusted according to the SLAM navigation control layer and the quality work order data, so as to generate a navigation execution strategy for the current quality re-inspection task based on the multimodal large model. Based on the navigation execution strategy, the industrial vision robot is controlled to perform SLAM path generation, motion control, visual review during the journey, and industrial vision inspection after arriving at the inspection point.

[0007] Furthermore, to achieve the above objectives, the present invention also provides an industrial vision robot navigation control system based on multimodal large model and SLAM collaboration, comprising: A module is established to acquire key task data of the industrial vision robot in the current quality re-inspection task and establish robot navigation task records; wherein, the key task data includes map object data, quality work order data and production map data collected by the industrial vision robot; The conversion module is used to convert the map object data into semantic map object units for the robot navigation task record, and to determine the quality impact data and process association data corresponding to each semantic map object unit based on the quality work order data and the production map data. The determination module is used to determine the map retention status of each semantic map object unit in the current quality re-inspection task based on the quality impact data and the process association data. The adjustment module is used to generate a SLAM navigation control layer based on the map retention state, and adjust the inspection priority of the points to be inspected according to the SLAM navigation control layer and the quality work order data, so as to generate a navigation execution strategy for the current quality re-inspection task based on the multimodal large model. The control module is used to control the industrial vision robot to perform SLAM path generation, motion control, visual review during the journey, and industrial vision inspection after arriving at the inspection point, according to the navigation execution strategy.

[0008] The beneficial effects of this invention are as follows: It proposes an industrial vision robot navigation control method and system based on multimodal large model and SLAM collaboration. By introducing quality work order data and production map data into the process of determining the object retention state of the semantic map, the industrial vision robot can determine whether temporary objects on site are related to the current quality re-inspection task, and control the influence of objects in the SLAM navigation control layer accordingly. By using the map retention state for inspection priority adjustment, the robot can not only avoid or re-inspect related objects, but also prioritize reaching positions with higher quality disposal value. By updating the quality work order and production map state through industrial vision inspection results, subsequent navigation control can be adjusted accordingly with the quality re-inspection results, thereby improving the adaptability and accuracy of robot re-inspection navigation in flexible manufacturing production lines. Attached Figure Description

[0009] Figure 1 This is a flowchart illustrating the industrial vision robot navigation and control method based on multimodal large model and SLAM collaboration proposed in this embodiment of the invention. Figure 2 This is a schematic diagram illustrating the execution process of the industrial vision robot navigation and control method based on multimodal large model and SLAM collaboration proposed in this embodiment of the invention. Figure 3 This is a schematic diagram illustrating the principle of semantic map object retention and map state transition proposed in this embodiment of the invention; Figure 4 This is a schematic diagram illustrating the execution process of detection result feedback update and subsequent navigation adjustment proposed in this embodiment of the invention; Figure 5This is a schematic diagram of the structure of the industrial vision robot navigation control system based on multimodal large model and SLAM collaboration proposed in this embodiment of the invention. Detailed Implementation

[0010] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0011] This invention provides a navigation and control method for industrial vision robots based on multimodal large model and SLAM collaboration, referring to... Figures 1-2 In this embodiment, an industrial vision robot navigation and control method based on multimodal large model and SLAM collaboration includes the following steps: S1: Obtain key task data of the industrial vision robot in the current quality re-inspection task and establish a robot navigation task record; wherein, the key task data includes map object data, quality work order data and production map data collected by the industrial vision robot.

[0012] Specifically, after receiving a quality re-inspection task, the industrial vision robot first acquires the key task data required for the current task. This key task data includes map object data, quality work order data, and production map data. The map object data is obtained by the industrial vision robot during its on-site operation using an industrial camera, depth sensor, or in conjunction with a SLAM module. It describes objects appearing within the robot's field of view or a local map area. The map object data includes at least the object number, object category, object location, object occupied area, object appearance time, and object most recently observed time. The object category can be a rework pallet, temporary material cart, changeover fixture, temporary inspection table, barrier, fixed equipment, passageway marker, or other objects that can affect robot passage and re-inspection operations. The object location can be represented using SLAM map coordinates, and the object occupied area can be represented using a grid area, rectangular area, circular area, or polygonal area; this invention does not limit this.

[0013] Quality work order data describes the handling requirements of the current quality re-inspection task. Specifically, quality work order data can come from a quality management system, manufacturing execution system, or industrial IoT platform, and it includes at least the work order number, defect category, defect severity, batch involved, process to be re-inspected, handling status, and re-inspection priority. For example, if the current work order records "Batch B20260524 has a burr defect at the crimping station, requiring re-inspection of the crimping station and its downstream assembly station," then the defect category is crimping burr, the batch involved is B20260524, the processes to be re-inspected include the crimping station and the assembly station, and the handling status can be pending re-inspection or re-inspection in progress.

[0014] Production map data is used to describe the relationships between processes, equipment, tooling, and product flow in a production line. Specifically, production map data includes at least process nodes, equipment nodes, tooling usage relationships, relationships between preceding and subsequent processes, and defect propagation relationships. The production map is not limited to a specific database format; it can use graph databases, structured tables, JSON structures, or other data formats capable of expressing process relationships. In this embodiment of the invention, the production map's function is not simply to display the production line structure, but rather to determine the degree of process association between map objects and the current quality re-inspection task. For example, a relationship chain can be formed between the crimping station, crimping tool storage point, rework buffer area, and re-inspection station in the production map. When the robot discovers a rework tray or temporary fixture near these areas, the system can further determine whether the object may be related to the current quality work order.

[0015] In this embodiment of the invention, after acquiring the aforementioned key task data, it is written into the robot navigation task record according to the same task number. The robot navigation task record can be understood as a data carrier that runs through the current navigation control process, used to record map objects, quality work orders, production maps, semantic map object status, inspection priorities, navigation strategies, detection results, and task completion results.

[0016] In one executable implementation, the robot navigation task record may include a task number, the robot's current position, the current SLAM map number, the quality work order number, the production map version number, the current set of points to be inspected, and a list of map objects. For example, in a crimping burr re-inspection task, the robot starts from the workshop entrance. The current SLAM map number is M-01, the quality work order number is Q-20260524-01, the production map version number is G-05, and the set of points to be inspected includes the crimping station, the rework buffer area, and the downstream assembly station. When the robot identifies a rework tray and a temporary fixture during its movement, it adds the corresponding objects to the map object list and records the object's appearance time and the object's most recent observation time.

[0017] S2: For the robot navigation task record, the map object data is converted into semantic map object units, and the quality impact data and process association data corresponding to each semantic map object unit are determined according to the quality work order data and the production map data.

[0018] Specifically, after establishing the robot navigation task record, the system converts map object data into semantic map object units. These semantic map object units include not only the object's location and occupied area on the map, but also the object's category, appearance time, most recent observation time, relationship between the object and the passageway, relationship between the object and inspection points, and the object's initial map retention state. Through this conversion process, objects obtained through visual recognition can be transformed into map objects that participate in navigation control.

[0019] In this embodiment of the invention, the conversion process first maps the object to a corresponding area in the SLAM map based on the object's location and occupied area in the map object data. For example, an industrial vision robot identifies a rework tray near a crimping station. Its visual detection results provide the tray's orientation and distance relative to the robot's current position. The SLAM module converts this result into SLAM map coordinates based on the robot's current pose, forming a corresponding occupied area. Subsequently, the system reads the object's category, occurrence time, and most recent observation time, and merges this information with the SLAM map coordinates to form a semantic map object unit.

[0020] Furthermore, the system determines quality impact data based on quality work order data. This quality impact data represents the degree of influence of the current quality work order on the navigation control process. In this embodiment, the quality impact data can be determined based on defect severity, re-inspection priority, and handling status. In one execution mode, it can be represented by the product of the defect severity coefficient, re-inspection priority coefficient, and handling status coefficient. Specifically, the higher the defect severity of a quality work order, the larger the defect severity coefficient; the higher the re-inspection priority coefficient of a quality work order, the greater its re-inspection priority; when the work order is in an incomplete, pending re-inspection, or re-inspection state, the handling status coefficient is larger, while when the work order is closed or confirmed as completed, the handling status coefficient is smaller.

[0021] Furthermore, the system determines process association data based on production map data. Specifically, for each semantic map object unit, the system determines the process node or equipment node corresponding to the object's location, and searches the map distance and process relationship strength between the node and the process nodes involved in the quality work order based on the production map. The map distance is used to represent the proximity of two process nodes in the production relationship, and the process relationship strength is used to represent whether there is a relationship such as defect formation, defect transmission, re-inspection dependence, or tooling sharing between two processes. In this embodiment of the invention, process association data can be represented as: ; in, Indicates the first The semantic map object unit and the first Process association data between quality work orders; Indicates the first The process node where the semantic map object unit is located is related to the first Each quality work order involves the spectral distance between process nodes; Indicates the first The process node where the semantic map object unit is located is related to the first A quality work order involves the strength of the process relationships between process nodes. It's easy to understand that the closer the process of an object is to the processes involved in the quality work order, and the stronger the process relationship, the higher the degree of association between that object and the current quality work order.

[0022] In practical applications, assuming the current quality work order involves the crimping station, and the robot finds a rework tray in the rework buffer, and the production map shows a defect re-inspection relationship between the rework buffer and the crimping station, then the rework tray corresponds to... Higher. Conversely, if the robot finds a regular material cart in a general logistics channel far from the process involved in the work order, then the corresponding regular material cart... The importance is relatively low. In this way, the system can avoid judging the importance of an object solely based on its category, and instead make a judgment based on the object's location and the production relationship between the processes involved in the quality work order.

[0023] It should be noted that the semantic map object unit in this invention is not limited to a specific data table structure. As long as it can simultaneously express the map location of the object, the object category, the time attribute, and the data relationship related to the quality re-inspection task, it can be used as an implementation of this invention.

[0024] S3: Based on the quality impact data and the process association data, determine the map retention status of each semantic map object unit in the current quality re-inspection task.

[0025] Specifically, after obtaining the semantic map object units, quality impact data, and process association data, the system further determines the retention rate of each semantic map object unit in the current quality re-inspection task. The retention rate indicates the degree to which a map object should still be maintained by the map and influence robot navigation control during the current navigation control process. A higher retention rate indicates that the object is less likely to be released immediately; a lower retention rate indicates that the object is more likely to no longer affect the current task, and its impact on navigation control can be gradually reduced.

[0026] In traditional processing, temporary objects are typically retained based on fixed time intervals. For example, ordinary material carts are set to be removed after ten minutes, and isolation barriers are set to be removed after thirty minutes. This method is relatively simple in ordinary scenarios, but it is insufficient in quality re-inspection scenarios. This is because even the same rework pallet might be just an ordinary temporary object when there is no quality work order, but could be a re-inspection object or a rework object when the current work order involves the corresponding batch. Therefore, this invention does not determine retention solely based on object category and time, but instead uses both quality impact data and process correlation data in the retention calculation.

[0027] In this embodiment of the invention, the basic retention time is first determined based on the object category of the semantic map object unit. The basic retention time can be configured according to experience or on-site management rules. For example, the basic retention time for fixed equipment is longer, the basic retention time for ordinary material carts is shorter, and the basic retention time for rework pallets and temporary inspection stations can be at an intermediate level. Then, the system determines the duration of no further observation based on the object's most recent observation time, that is, the difference between the current moment and the time when the object was last visually confirmed by the robot.

[0028] Furthermore, the system determines quality correlation enhancement data based on quality impact data, process correlation data, and the matching coefficient between object category and quality re-inspection task. The quality correlation enhancement data can be represented as: ; in, Indicates the first A semantic map object unit at time... Quality-related data enhancement; This indicates the number of valid quality work orders in the current robot navigation task record; Indicates the first Quality impact data corresponding to each quality work order; Indicates the first The semantic map object unit and the first Process association data between quality work orders; Indicates the first The matching coefficient between the object category of each semantic map object unit and the quality re-inspection task; This represents the natural logarithm operation. Therefore, the amount of data enhancing quality correlation increases with the degree of quality correlation, while avoiding the unreasonable long-term retention of objects due to excessively high values ​​of a single data point.

[0029] Furthermore, the system determines clearance-promoting data based on visual confirmation clearance data, access restoration data, and the matching coefficient between object categories and quality re-inspection tasks. In one implementation method, the clearance-promoting data can be: Characterization; among which, Indicates the first A semantic map object unit at time... The clearing process facilitates data processing; This indicates that industrial vision robots are at any time For the Each semantic map object cell confirms that it has been moved, and visual confirmation clears the data; Indicates the first The region where each semantic map object cell is located at time [time] Traffic recovery data; Indicates the first The matching coefficient between the object category of each semantic map object unit and the quality re-inspection task. When the robot re-observes the area and confirms that the object has been moved, and the SLAM map shows that the area is passable, and the object has a low degree of matching with the current quality re-inspection task, the object's retention rate should be reduced more quickly.

[0030] Based on the above data, the system determines the retention rate, which can be expressed as: ; in, Indicates the first A semantic map object unit at time... Retention rate; Indicates the current time; Indicates the first The most recent observation time of each semantic map object unit; Represents object category The corresponding base retention time; This indicates the removal of promotional data; This indicates data with enhanced quality correlation; This represents an exponential operation. When an object is not observed again for a long time, its retention rate tends to decrease; when an object has a strong relationship with the current quality work order and production map, the quality correlation enhancement data increases, slowing down the rate of retention rate decrease; when an object is visually confirmed to have been moved and access has been restored, the removal promotion data increases, accelerating the rate of retention rate decrease.

[0031] In this embodiment of the invention, the system compares the retention rate with a preset threshold and determines the map retention status accordingly. Specifically, when the retention rate is greater than or equal to a third threshold... When the retention rate is at the second threshold, it is determined to be a strong retention state; when the retention rate is at the second threshold... With the third threshold When the retention rate is between these thresholds, it is determined to be in a weak retention state; when the retention rate is at the first threshold... With the second threshold When the retention rate is between [a certain threshold], it is determined to be in a pending confirmation state; when the retention rate is less than the first threshold... When a task is completed, it is determined to be in a removable state. The above thresholds can be configured based on workshop aisle width, robot size, quality work order processing time limits, and on-site safety requirements. For example, a strong retention state can be used for rework pallets that are highly relevant to the current work order and may still occupy the aisle; a weak retention state can be used for ordinary material carts that may still exist but do not completely block the aisle; a pending confirmation state can be used for temporary fixtures that have not been observed again for a long time but have not yet been confirmed to have been removed; and a removable state can be used for objects that have been visually confirmed to have been removed and have a low correlation with the current work order.

[0032] like Figure 3 As shown, in this embodiment of the invention, the retention rate of semantic map objects is jointly determined by the basic retention time, the most recent observation time, quality impact data, process association data, visual confirmation clearing data, and access restoration data. Specifically, if an object has a strong relationship with related processes in the current quality work order and production map, its quality association degree is high, and the retention rate of the object decreases slowly; if an object has been confirmed by robot vision to have been moved, and the corresponding area has been restored to accessibility, the retention rate of the object decreases quickly. The system compares the retention rate with multiple thresholds, and then classifies the object into a strong retention state, a weak retention state, a pending confirmation state, or a clearable state, and converts them into a strong restriction area, a slow-moving area, a review area, or a release area in the SLAM navigation control layer, respectively.

[0033] Through step S3, the retention status of map objects is simultaneously affected by object category, quality work order, production map and visual confirmation results. Thus, the present invention can adapt to the characteristic that the meaning of temporary objects changes with the task in the quality re-inspection scenario of flexible production line.

[0034] S4: Generate a SLAM navigation control layer based on the map retention state, and adjust the inspection priority of the points to be inspected according to the SLAM navigation control layer and the quality work order data, so as to generate a navigation execution strategy for the current quality re-inspection task based on the multimodal large model.

[0035] Specifically, after determining the map retention state of each semantic map object unit, the system converts the map retention state into a SLAM navigation control layer. This SLAM navigation control layer provides the SLAM module with area information that can be used for path generation and motion control. In this embodiment, a strong retention state corresponds to a strongly restricted area, a weak retention state corresponds to a slow-moving area, a pending confirmation state corresponds to a review area, and a clearable state corresponds to a released area. A strongly restricted area indicates an area that the robot must not enter or needs to replan its route to avoid; a slow-moving area indicates that the robot can pass through at a low speed while maintaining a safe distance; a review area indicates that visual review needs to be triggered when the robot passes nearby; and a released area indicates that the object no longer restricts the current navigation control.

[0036] When generating the SLAM navigation control layer, the system can determine the influence range of an object based on its occupied area and retention rate. The influence range of each semantic map object unit can be characterized by calculating the product of the conversion coefficient from retention rate to safe extension distance and the retention rate of the corresponding semantic map object unit at the corresponding time, and then summing this product with the basic geometric range of the corresponding semantic map object unit. Therefore, when the object retention rate is high, its influence range on the SLAM map increases accordingly; when the object retention rate decreases, its influence range gradually decreases. In this way, the influence of map objects on robot passage can change smoothly with variations in retention rate, avoiding path execution instability caused by sudden changes in object state.

[0037] Furthermore, the system adjusts the inspection priority of the points to be inspected based on the SLAM navigation control layer and quality work order data. These points can include defect-occurring workstations, upstream workstations, downstream workstations, rework buffer areas, tooling storage areas, and temporary inspection stations. First, the system determines the basic correlation between the points to be inspected and the current quality re-inspection task based on the quality work order data. For example, the basic correlation is high for crimping workstations directly involved in the quality work order, followed by downstream assembly workstations that may be related to defect transmission, and low for ordinary areas that have no obvious relation to the current work order.

[0038] Subsequently, the system determines the map impact based on the retention rate of semantic map object units within the vicinity of the inspection point and the matching coefficient between the object categories and the quality re-inspection task. In one execution mode, the map impact of each inspection point can be represented by calculating the cumulative sum of the matching coefficients between the object categories and the quality re-inspection task of all semantic map object units and the retention rate at the corresponding time. If there are multiple strongly retained objects related to the quality re-inspection task around an inspection point, the map impact of that inspection point is high, indicating that the area has higher attention value for the current quality handling.

[0039] Furthermore, the system determines the passage cost based on the robot's current position, the SLAM navigation control layer, and the location of the inspection point. The passage cost can comprehensively consider path length, the degree of detour through heavily restricted areas, the number of slow-moving areas, the number of areas requiring re-inspection, and the time required for the robot to reach the inspection point. Then, the system determines the inspection priority based on basic relevance, map influence, and passage cost. The inspection priority can be expressed as: ; in, Indicates the first Inspection priority of each point to be inspected; Indicates the first The basic correlation between the points to be inspected and the current quality re-inspection tasks; Indicates the first The map impact of each inspection point; This indicates that the industrial vision robot has reached the [number]th position from its current position. The access cost to each inspection point. The more relevant an inspection point is to the quality work order, and the more its surrounding objects indicate that the area is related to the current quality action, the higher the priority of that point; however, if the access cost to reach that point is too high, the priority will be appropriately reduced.

[0040] After determining the inspection priority, the multimodal large model reads the robot navigation task record, SLAM navigation control layer, and inspection priority, and generates a navigation execution strategy. It should be noted that the multimodal large model in this invention does not directly control the robot chassis motors, but rather organizes the current task semantics, map state, and quality re-inspection requirements into an executable navigation strategy. In this embodiment, when the multimodal large model organizes the current task semantics, map state, and quality re-inspection requirements into an executable navigation strategy, it does not directly output open natural language control commands, but rather generates a structured strategy based on pre-defined navigation strategy field templates and field value rules.

[0041] It should be noted that the pre-set navigation strategy field templates and field value rules are preferably manually pre-configured engineering rules, configured by system implementers based on the navigation actions that the industrial vision robot can perform, the types of areas that the SLAM module can recognize, the speed levels that the motion control module can accept, and the detection actions that the industrial vision inspection module can perform. Specifically, the navigation strategy field template includes at least the following fields: next inspection point, passage mode, speed level, en route review requirements, arrival inspection requirements, and inspection result feedback requirements. The field value rules are used to limit the selectable values ​​of each field. For example, the passage mode can be limited to bypassing strongly restricted areas, passing through slow-moving areas at low speed, stopping for review after passing through a review area, and passing normally through a released area. The speed level can be limited to normal speed, low speed, and stopping and waiting. The arrival inspection requirements can be limited to defect area photography, batch label recognition, tooling status confirmation, or equipment status recognition. Through the above-mentioned manual pre-configuration method, the output range of the multimodal large model is limited to the structured fields that the robot system can execute and verify, thereby avoiding the generation of unexecutable or ambiguous control instructions.

[0042] Specifically, the system first organizes the quality work order number, defect category, re-inspection process, set of inspection points, inspection priority of each inspection point, map retention status of semantic map object units, and strongly restricted areas, slow-moving areas, re-inspection areas, and release areas in the SLAM navigation control layer from the robot navigation task record into a structured input that can be read by the multimodal large model. Among them, the quality work order number and defect category are used to limit the re-inspection purpose of this navigation task, the set of inspection points and inspection priority are used to limit the candidate arrival positions, and the map retention status and SLAM navigation control layer are used to limit the robot's travel mode and safety constraints during the movement. During the generation process, the multimodal large model is filled in sequentially according to the above navigation strategy field template, including the next inspection point, travel mode, speed level, re-inspection requirements along the way, inspection requirements after arrival, and inspection result feedback requirements, and outputs the above content in the form of structured instructions. For example, when the inspection priority ranking result shows that the crimping re-inspection position has the highest priority, and there are strong restriction areas corresponding to the rework tray and re-inspection areas corresponding to the temporary fixture near its passage path, the navigation strategy generated by the multimodal large model can include: the next inspection point is the crimping re-inspection position; the passage method is to bypass the strong restriction area corresponding to the rework tray; the speed level is to travel at normal speed in the ordinary channel and reduce speed after entering the slow-moving area; the re-inspection requirement during the journey is to stop, collect images, and confirm the object status when passing the re-inspection area corresponding to the temporary fixture; the inspection requirement after arrival is to photograph the crimping terminal area and identify the batch label; the inspection result feedback requirement is to write the defect inspection result, batch identification result, and re-inspection object status into the robot navigation task record.

[0043] Furthermore, after receiving the output of the multimodal large model, the system performs format and existence checks on the inspection point number, region type, speed level, and detection action in the structured instruction. When all output fields can be found in the robot navigation task record, SLAM navigation control layer, or preset action set, the structured instruction is sent to the SLAM module and motion control module for execution. When there are unrecognizable inspection points, region types, or action fields in the output, the system re-requests the generation of the multimodal large model based on the current inspection priority ranking result, or selects the inspection point with the highest priority and a passable path as the next inspection point. Thus, the role of the multimodal large model is limited to semantic organization and strategy generation of structured task data under the constraints of manually pre-configured field templates and executable action sets.

[0044] In one optional implementation, to improve the adaptability of the multimodal large model to different factory expression habits, work order description methods, and on-site object names, a prompt example library can be constructed based on historical quality work orders, historical inspection records, manually confirmed navigation strategy samples, and on-site terminology. This allows the multimodal large model to refer to historical samples similar to the current task when generating navigation strategies. Alternatively, the aforementioned historical samples can be used as model fine-tuning corpus, making it easier for the model to output structured navigation strategies according to the field templates defined in this invention. It should be noted that the prompt example library or model fine-tuning is only an optional method to improve generation stability and is not a necessary condition for implementing this invention. Even without model training, those skilled in the art can still realize the process of generating executable navigation strategies using the multimodal large model in this invention by manually configuring navigation strategy field templates, field value rules, action sets, and output verification rules.

[0045] S5: Based on the navigation execution strategy, control the industrial vision robot to perform SLAM path generation, motion control, visual review during the journey, and industrial vision inspection after arriving at the inspection point.

[0046] Specifically, after generating a navigation execution strategy from the multimodal large model, the SLAM module generates an executable path for the robot based on this strategy. The executable path includes at least a travel path, turning position, stopping position, slow-moving area, re-inspection position, and arrival posture requirements. The motion control module controls the movement of the industrial vision robot according to the executable path. When the robot passes through a slow-moving area, the motion control module reduces its speed and maintains a safe distance; when the robot passes through a re-inspection area, the motion control module controls the robot to decelerate or pause briefly, triggering the industrial vision inspection module to re-inspect relevant semantic map objects; when the robot encounters a strongly restricted area, the SLAM module does not treat this area as a passable area, but instead generates a path to bypass it.

[0047] During the en-route visual review, the industrial vision inspection module re-inspects semantic map objects in the unconfirmed or weakly retained state and writes the results into the robot navigation task record. If the review result shows that the object still exists, the system updates the object's most recent observation time and continues to retain the object's influence in subsequent calculations. If the review result shows that the object has been moved, the system increases the visual confirmation to clear the data, making it easier for the object to enter the removable state in subsequent retention calculations. If the review result shows that the object category has changed, for example, a regular pallet is identified as a return pallet with the current batch label, the system updates the matching coefficient between the object category and the quality re-inspection task, potentially changing the object from a weakly retained state to a strongly retained state.

[0048] Once the industrial vision robot arrives at the inspection point, the industrial vision inspection module performs inspections according to the arrival inspection requirements in the navigation execution strategy. Inspection content may include defect area imaging, batch label recognition, tooling status confirmation, equipment indicator status recognition, product placement status confirmation, and abnormal object evidence collection. The inspection results include at least the inspection point number, inspection time, inspection image number, presence of defects, defect type, defect severity, batch identification result, inspection reliability, and inspection conclusion. It should be noted that this invention does not limit the specific visual recognition network or image processing method used; any method that can output the above inspection results and write them into the robot navigation task record can be considered an optional implementation of this invention.

[0049] Furthermore, the system updates quality work order data and production map data based on the inspection results. When the inspection results indicate that the anomaly persists, the system can increase the handling status coefficient in the quality work order, maintain or increase the re-inspection priority, and increase the re-inspection attention of the corresponding process node in the production map. When the inspection results indicate that the anomaly has been resolved, the system can decrease the handling status coefficient in the quality work order, decrease the re-inspection attention of the relevant process node, and make it easier for temporary objects associated with the work order to enter a weakly retained state or a clearable state. The update of the re-inspection attention of nodes in the production map can be characterized by summing the re-inspection attention of nodes in the production map at the previous moment, the increase in attention when the inspection results indicate that the anomaly still exists, and the decrease in attention when the inspection results indicate that the anomaly has been resolved. Thus, the production map not only expresses static process relationships but also the changes in process attention during the current quality handling process.

[0050] like Figure 4As shown, in this embodiment of the invention, after the robot arrives at the inspection point, it performs industrial vision inspection and updates the subsequent navigation control process based on the inspection results. When the inspection results indicate that the anomaly still exists, the system updates the quality work order status and production map attention level, ensuring that the relevant process nodes and related map objects maintain a high level of attention. When the inspection results indicate that the anomaly has been resolved, the system correspondingly reduces the quality work order handling intensity and production map attention level, reducing the retention rate of the relevant objects in subsequent calculations. Subsequently, the system recalculates the object retention rate, re-determines the map retention status, regenerates the SLAM navigation control layer, and reorders the priority of the inspection points, ultimately forming the updated next inspection point and navigation path.

[0051] After completing the above updates, the system re-determines the quality impact data, process association data, semantic map object retention rate, map retention status, SLAM navigation control layer, and inspection priority. Therefore, the industrial vision inspection results can further influence the subsequent navigation control process. For example, if the robot detects a defect still existing at the crimping station, the re-inspection focus on the crimping station, rework buffer area, and related tooling areas will increase, and the retention rate of related rework trays and temporary inspection stations may remain high. If the robot confirms that the defect has been resolved and the rework tray has been removed, the retention rate of related objects will decrease, the SLAM navigation control layer will release the corresponding area, and the robot's subsequent route can return to normal.

[0052] Upon completion of the current quality re-inspection task, the system outputs a navigation task completion record. This record can include the final retained objects, cleared objects, objects requiring manual confirmation, the robot's route, inspection results at each inspection point, changes in the quality work order status, and changes in the re-inspection focus of production map nodes. Final retained objects can be marked with reasons for retention, such as: related to incomplete quality work orders, located near defect re-inspection processes, or still occupying a passageway. Cleared objects can be marked with reasons for removal, such as: visual confirmation that they have been moved, the quality work order has been closed, or the production map association has decreased. Objects requiring manual confirmation can generate prompts and send them to the field terminal.

[0053] In a specific application example, after a product changeover is completed in the morning on an electronic assembly line, the quality system generates a crimping burr re-inspection work order, requiring the robot to inspect the crimping station, rework buffer, and downstream assembly station. When the robot departs from the workshop entrance, it identifies a regular material cart and a rework pallet. Although the regular material cart occupies part of the passageway, its relationship with the current quality work order is weak, so its retention level gradually decreases, and it is written into the slow-moving area. The rework pallet is located near the crimping station; the production map shows that it is related to the crimping re-inspection, and the quality work order has not yet been completed. Therefore, the rework pallet is identified as having a strong retention status, and its surrounding area is written into a strong restriction area. Subsequently, the system increases the inspection priority of the crimping re-inspection station and the rework buffer. The robot first bypasses the rework pallet to reach the crimping re-inspection station and completes the defect detection. If the inspection result shows that the burr still exists, the system maintains a high retention level for the relevant objects and arranges for the robot to continue to the rework buffer for re-inspection; if the inspection result shows that the anomaly has been resolved and the rework pallet has been removed, the system releases the corresponding area, allowing subsequent inspection routes to return to normal. Therefore, this invention enables the robot to adjust the map retention and navigation control process in a timely manner according to the quality handling status.

[0054] Reference Figure 5 , Figure 5 This is a schematic diagram of the structure of an industrial vision robot navigation control system based on multimodal large model and SLAM collaboration according to an embodiment of the present invention.

[0055] like Figure 5 As shown, the industrial vision robot navigation control system based on multimodal large model and SLAM collaboration proposed in this embodiment of the invention includes: Module 10 is used to acquire key task data of the industrial vision robot in the current quality re-inspection task and establish a robot navigation task record; wherein, the key task data includes map object data, quality work order data and production map data; The conversion module 20 is used to convert the map object data into semantic map object units for the robot navigation task record, and to determine the quality impact data and process association data corresponding to each semantic map object unit based on the quality work order data and the production map data. The determination module 30 is used to determine the map retention status of each semantic map object unit in the current quality re-inspection task based on the quality impact data and the process association data. The generation module 40 is used to generate a SLAM navigation control layer based on the map retention state, and adjust the inspection priority of the points to be inspected according to the SLAM navigation control layer and the quality work order data, so as to generate a navigation execution strategy for the current quality re-inspection task based on the multimodal large model. The control module 50 is used to control the industrial vision robot to perform SLAM path generation, motion control, visual review during the journey, and industrial vision inspection after arriving at the inspection point, according to the navigation execution strategy.

[0056] Other embodiments or specific implementations of the industrial vision robot navigation control system based on multimodal large model and SLAM collaboration of the present invention can be referred to the above-mentioned method embodiments, and will not be repeated here.

[0057] It is understood that in the description of this specification, references to terms such as "one embodiment," "another embodiment," "other embodiments," or "first embodiment to Nth embodiment," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0058] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0059] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A navigation and control method for industrial vision robots based on multimodal large model and SLAM collaboration, characterized in that, The method includes the following steps: Acquire key task data of the industrial vision robot in the current quality re-inspection task and establish a robot navigation task record; wherein, the key task data includes map object data, quality work order data and production map data collected by the industrial vision robot; For the robot navigation task record, the map object data is converted into semantic map object units, and the quality impact data and process association data corresponding to each semantic map object unit are determined based on the quality work order data and the production map data. Based on the quality impact data and the process association data, determine the map retention status of each semantic map object unit in the current quality re-inspection task; A SLAM navigation control layer is generated based on the map retention state, and the inspection priority of the points to be inspected is adjusted according to the SLAM navigation control layer and the quality work order data, so as to generate a navigation execution strategy for the current quality re-inspection task based on the multimodal large model. Based on the navigation execution strategy, the industrial vision robot is controlled to perform SLAM path generation, motion control, visual review during the journey, and industrial vision inspection after arriving at the inspection point.

2. The industrial vision robot navigation and control method based on multimodal large model and SLAM collaboration as described in claim 1, characterized in that, Acquire key task data of the industrial vision robot in the current quality re-inspection task, and establish a robot navigation task record, specifically including: Acquire map object data identified by the industrial vision robot in the current area, quality work order data corresponding to the current quality re-inspection task, and production map data corresponding to the quality work order data; The map object data, the quality work order data, and the production map data are written into the robot navigation task record according to the same task number.

3. The industrial vision robot navigation and control method based on multimodal large model and SLAM collaboration as described in claim 1, characterized in that, For the robot navigation task record, the map object data is converted into semantic map object units, specifically including: Using the object location and object-occupied area in the map object data, the objects identified by the industrial vision robot are mapped to the corresponding areas in the SLAM map to obtain the spatial occupancy results of the objects in the SLAM map; Based on the object category, object occurrence time, and object most recent observation time in the map object data, configure semantic and temporal attributes for the corresponding objects; A semantic map object unit is generated based on the spatial occupancy result, the semantic attribute, and the temporal attribute; wherein, the semantic map object unit includes object number, object category, object location, object occupied area, object appearance time, object most recent observation time, channel influence relationship, inspection point influence relationship, and map retention status; Write the semantic map object unit into the robot navigation task record.

4. The industrial vision robot navigation and control method based on multimodal large model and SLAM collaboration as described in claim 3, characterized in that, Based on the quality work order data and the production map data, the quality impact data and process association data corresponding to each semantic map object unit are determined, specifically including: Based on the defect severity, re-inspection priority, and handling status in the quality work order data, determine the quality impact data; Based on the production map data, determine the map distance and process relationship strength between the process node where each semantic map object unit is located and the process node involved in the quality work order; Based on the distance in the graph and the strength of the process relationship, process association data is determined; The quality impact data and the process association data are written into the robot navigation task record, and a corresponding relationship is established with the corresponding semantic map object unit.

5. The industrial vision robot navigation and control method based on multimodal large model and SLAM collaboration as described in claim 1, characterized in that, Based on the quality impact data and the process association data, the map retention status of each semantic map object unit in the current quality re-inspection task is determined, specifically including: The basic retention time is determined based on the object category of the semantic map object unit, and the duration of no re-observation is determined based on the object's most recent observation time. Based on the quality impact data, the process association data, and the matching coefficient between the object category and the quality re-inspection task, quality association enhancement data is determined; The retention rate is calculated based on the baseline retention time, the duration without re-observation, the quality correlation enhancement data, and the clearance promotion data. The retention rate is compared with a preset threshold, and the semantic map object unit is divided into a strong retention state, a weak retention state, a pending confirmation state, or a clearable state based on the comparison result.

6. The industrial vision robot navigation and control method based on multimodal large model and SLAM collaboration as described in claim 5, characterized in that, Based on the map's preserved state, a SLAM navigation control layer is generated, specifically including: Write the semantic map object units corresponding to the strongly preserved state into the strongly restricted area of ​​the SLAM navigation control layer, write the semantic map object units corresponding to the weakly preserved state into the slow-moving area of ​​the SLAM navigation control layer, write the semantic map object units corresponding to the pending confirmation state into the review area of ​​the SLAM navigation control layer, and write the semantic map object units corresponding to the clearable state into the release area. The scope of an object's influence is determined based on the object's occupied area and retention rate within the semantic map object unit; The scope of influence of the object is written into the SLAM navigation control layer so that the SLAM module performs path generation and motion control based on the SLAM navigation control layer.

7. The industrial vision robot navigation and control method based on multimodal large model and SLAM collaboration as described in claim 6, characterized in that, The inspection priority of the points to be inspected is adjusted based on the SLAM navigation control layer and the quality work order data, so as to generate a navigation execution strategy for the current quality re-inspection task based on the multimodal large model, specifically including: Determine the basic relevance between the points to be inspected and the current quality re-inspection tasks based on the quality work order data; The map impact is determined based on the retention rate of semantic map object units in the vicinity of the inspection point and the matching coefficient between the object category and the quality re-inspection task. Based on the robot's current position, the SLAM navigation control layer, and the location of the point to be inspected, calculate the travel cost for the robot to reach the point to be inspected; Based on the basic relevance, the map influence, and the passage cost, the inspection priority of the points to be inspected is generated.

8. The industrial vision robot navigation and control method based on multimodal large model and SLAM collaboration as described in claim 1, characterized in that, Based on the navigation execution strategy, the industrial vision robot is controlled to perform SLAM path generation, motion control, visual review during the journey, and industrial vision inspection upon arrival at the inspection point, specifically including: The robot navigation task record, SLAM navigation control layer, and inspection priority are input into the multimodal large model to obtain the navigation execution strategy output by the multimodal large model; wherein, the navigation execution strategy includes the next inspection point, the mode of travel, the speed level, the requirements for re-inspection along the way, and the requirements for detection after arrival; The navigation execution strategy is sent to the SLAM module so that the SLAM module can generate an executable path for the robot based on the next inspection point and the SLAM navigation control layer. The robot's executable path is sent to the motion control module to control the industrial vision robot to move to the corresponding position according to the speed level, passage mode, and on-the-way inspection requirements; After the industrial vision robot arrives at the next inspection point, it performs industrial vision inspection according to the inspection requirements after arrival, and writes the inspection results into the robot navigation task record.

9. The industrial vision robot navigation and control method based on multimodal large model and SLAM collaboration as described in claim 8, characterized in that, The method further includes: Update the handling status, defect severity, or re-inspection priority in the quality work order data based on the inspection results, and redetermine the quality impact data based on the updated quality work order data; Update the re-inspection focus of the corresponding process nodes in the production map data based on the test results; Based on the updated quality impact data and the updated production map data, the retention rate of semantic map object units, map retention status, SLAM navigation control layers, and inspection priorities are redefined. When the current quality re-inspection task is completed, output a navigation task completion record.

10. A navigation and control system for industrial vision robots based on multimodal large model and SLAM collaboration, characterized in that, The system includes: A module is established to acquire key task data of the industrial vision robot in the current quality re-inspection task and establish robot navigation task records; wherein, the key task data includes map object data, quality work order data and production map data collected by the industrial vision robot; The conversion module is used to convert the map object data into semantic map object units for the robot navigation task record, and to determine the quality impact data and process association data corresponding to each semantic map object unit based on the quality work order data and the production map data. The determination module is used to determine the map retention status of each semantic map object unit in the current quality re-inspection task based on the quality impact data and the process association data. The adjustment module is used to generate a SLAM navigation control layer based on the map retention state, and adjust the inspection priority of the points to be inspected according to the SLAM navigation control layer and the quality work order data, so as to generate a navigation execution strategy for the current quality re-inspection task based on the multimodal large model. The control module is used to control the industrial vision robot to perform SLAM path generation, motion control, visual review during the journey, and industrial vision inspection after arriving at the inspection point, according to the navigation execution strategy.

Images