Robot operation system based on visual analysis

Abstract

The invention discloses a robot operation system based on visual analysis, and the system comprises a map information input module which is used for obtaining the route information of a target region and generating a route network; the industrial robot is used for moving along a route in the target area and executing a work task; the road sign module is provided with a plurality of radio frequency tags which are respectively arranged at key nodes of the route network and are used for providing position identification information for the industrial robot; the server is used for generating advancing information according to the work tasks of the industrial robots; the advancing information comprises an advancing route and a time interval of passing through each key node; wherein the industrial robot is provided with a camera. In the technical scheme provided by the invention, when the industrial robot executes the work task in the target area, the advancing route and the time of passing through the key node are planned for each industrial robot.

Description

Technical Field

[0001] The present application relates to the technical field of industrial robot operation, and in particular, to a robot operation system based on visual analysis. Background Art

[0002] Smart Factory is an advanced model of modern manufacturing. It realizes automation, intelligence and efficiency of production process by integrating advanced technologies such as Internet of Things (IoT), artificial intelligence (AI), robotics and big data analysis. In smart factories, robots play a vital role, especially in the transportation and processing of workpieces. The production line and logistics system in the factory are almost completely controlled by robots, automation equipment and intelligent systems. Human intervention is reduced to a minimum, and the main task is to monitor and optimize the operation of the system. Robots are responsible for transporting raw materials, semi-finished products and finished products within the factory. Robots can autonomously plan paths, avoid obstacles and complete tasks efficiently based on real-time data and task requirements.

[0003] In an intelligent factory, the more robots there are, the higher the production efficiency of the intelligent factory. However, when there are too many robots, emergencies are inevitable, such as cargo dumping, personnel intrusion, robot movement failure, etc. At present, the common solution is to install a large number of cameras in the factory area, and in case of emergencies, timely control the surrounding robots to change routes or directions.

[0004] This control method is easily limited by blind spots in monitoring. When cargo blocks the camera's field of view, the accident area cannot be discovered in time, which in turn causes the accident level to expand. Summary of the invention

[0005] The content of this application is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this application is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0006] In order to solve the technical problems mentioned in the above background technology section, some embodiments of the present application provide a robot operation system based on visual analysis, including:

[0007] A map information input module is used to obtain route information of the target area and generate a route network;

[0008] Industrial robots, which are used to move along routes and perform work tasks within a target area;

[0009] The road sign module is equipped with multiple radio frequency tags, which are arranged at key nodes of the route network to provide location identification information to the industrial robot;

[0010] The server is used to generate travel information according to the work tasks of each industrial robot; the travel information includes the travel route and the time interval passing through each key node;

[0011] Among them, the industrial robot is equipped with a camera for adjusting the movement speed of the industrial robot in real time based on the image information collected by the camera.

[0012] In the technical solution provided by this application, when an industrial robot performs a work task in a target area, the server will plan the route and time to pass through key nodes for each industrial robot according to the requirements of the work task. This design can minimize the congestion of industrial robots at key locations, thereby reducing the probability of accidents. In addition, each industrial robot is integrated with a camera, which can adjust the robot's movement speed in real time according to the information fed back by the camera to avoid collisions.

[0013] Furthermore, the key node is a node where at least two routes intersect.

[0014] In the technical solution provided in the present application, by setting radio frequency tags at the intersections of the lines, the industrial robot can accurately understand its own position, and then when moving to the intersection, it can accurately choose the direction of travel, thereby increasing the success rate of task execution.

[0015] In the target area, the greater the density of industrial robots, the higher the moving speed, and the production efficiency usually increases accordingly. However, in order to achieve normal operation of industrial robots with greater density and higher moving speed, it is necessary to be able to control the industrial robots more timely. In the prior art, a two-way information interaction mechanism between industrial robots and servers is usually used to provide timeliness of industrial robot control. When an industrial robot detects an emergency, it will upload the information to the server, and the server will then issue instructions to all industrial robots based on the uploaded emergency information. However, in this process, the information needs to pass through multiple communication nodes, which inevitably causes delays. This delay may cause industrial robots to rear-end and other situations, resulting in an escalation of the accident level.

[0016] Furthermore, the industrial robot includes:

[0017] Execution and driving module, used to execute work tasks and travel information;

[0018] The first communication module is used to establish a signal connection with the server to obtain work tasks and travel information;

[0019] An image information extraction module, which is signal-connected to a camera, is used to extract the video sequence transmitted by the camera and extract road condition information therefrom.

[0020] An early warning module is used to generate early warning information based on road condition information and its own moving speed. The early warning information includes its own position, its own moving speed, its own acceleration, and obstacle area position information.

[0021] A second communication module is used to broadcast the early warning information outward through a high-frequency band.

[0022] A movement control module is used to generate a moving speed in real time based on road condition information, travel information, and early warning information. The moving speed includes a moving speed and a moving direction.

[0023] In the technical solution provided by this application, when the industrial robot moves, it first determines the path according to the travel route provided in the travel information, then determines the approximate moving speed according to the time interval for reaching each key node, and finally generates the final travel speed in combination with the road condition information. Therefore, when the server controls the movement of the industrial robot, it only needs to provide correct travel information, and there is no need to send a large number of control instructions for the movement of the robot. This design makes the movement control of the industrial robot not affected by network fluctuations. At the same time, the industrial robot can adjust its own speed in real time according to the road condition information during the movement process, thereby improving timeliness and reducing the influence of external interference. In addition, to avoid affecting the surrounding industrial robots, the industrial robot will send out early warning information through a high-frequency band, so that the nearby industrial robots can receive it and automatically plan a more reasonable travel speed. This mechanism maximally avoids congestion of industrial robots at key node positions, reduces the accident probability, improves the passing rate of key nodes, and thus improves production efficiency.

[0024] Due to the different travel speeds of industrial robots, it is inevitable that industrial robots need to overtake other industrial robots in front during the travel process. To avoid accidents during the overtaking process, the following technical solution is provided in this application:

[0025] The early warning information further includes information on whether the recent moving speed has changed.

[0026] In the technical solution provided by this application, the early warning information includes information on whether the recent moving speed has changed. Therefore, when an industrial robot needs to overtake an industrial robot in front, it can judge the overtaking time and position according to the recent moving speed change situation of the surrounding industrial robots, thereby improving the safety of the overtaking process.

[0027] Since each robot performs different tasks and different tasks have different priorities, it is necessary to increase the passing efficiency of these robots at key nodes. If different passing levels are directly assigned to each robot, when the passing level of a robot changes, the information needs to be sent to the remaining robots in a timely manner. In this way, it is difficult to change the passing level of the robot. When some robots do not receive the information in time, confusion may easily occur at key node locations. To this end, the present application provides the following technical solutions:

[0028] Furthermore, the server includes:

[0029] A work task input module is used to input the work tasks that need to be performed by each industrial robot and generate the execution time of each work task;

[0030] The work task allocation module allocates each work task to each industrial robot according to the execution time and generates corresponding travel information;

[0031] The priority adjustment module obtains the work tasks whose completion time limit needs to be adjusted, generates updated travel information based on the location of the target industrial robot; obtains the corresponding key nodes in the updated travel information, and sends control instructions to the key nodes; the control instructions include the travel direction within the corresponding time interval.

[0032] In the technical solution provided by the present application, the speed at which industrial robots perform tasks is regulated by adjusting the travel information of the industrial robots to achieve the completion time limit of the work tasks. At the same time, in order to avoid congestion of industrial robots at key nodes, the priority adjustment module will adjust the control instructions of key nodes according to the updated information, thereby ensuring that industrial robots with high priorities can quickly pass through key nodes, while industrial robots with low priorities wait at key nodes. This design allows the system to quickly and accurately complete work tasks and complete the time limit regulation without sending priority instructions to all industrial robots, thus avoiding confusion during the regulation process.

[0033] In the whole system, industrial robots are constantly moving, so when adjusting the work tasks, updated travel information and control instructions must be generated quickly. However, if these updated information are generated only for the purpose of meeting new work tasks, it is likely to interfere with the normal movement of other industrial robots in the system, thus hindering the smooth progress of the overall work.

[0034] Furthermore, the priority adjustment module includes:

[0035] Update the task acquisition unit to acquire the work tasks that need to adjust the work time limit;

[0036] An information base generation unit generates an adjustment plan matrix D based on the work tasks that require adjustment of the work time limit;

[0037] Among them, d 1 Indicates the first update progress information, d 2 Indicates the second update progress information; d i represents the i-th updated traveling information, i represents the index of the element in the adjustment scheme matrix D; the direction value K based on the updated traveling information in the adjustment scheme matrix D i Arrange by size;

[0038]

[0039] Where, j represents the index of the path in the i-th updated travel information, N i represents the number of paths in the i-th updated travel information, represents the vector of the jth path in the i-th updated travel information, The vector representing the industrial robot from its current position to the modified final position;

[0040] The particle swarm update diffusion unit is used to generate a number of particles in the adjustment scheme matrix D and perform iterative updates to find the best updated new information.

[0041] In the technical solution provided by the present application, the particle swarm algorithm is used to diffuse in the adjustment scheme matrix D, and the best adjustment scheme can be quickly found from a large number of adjustment scheme matrices. In order to better find the direction that is conducive to achieving the best adjustment during the particle diffusion process, the elements in the adjustment scheme matrix D are arranged according to the size of the direction value. Therefore, the elements in the adjustment scheme matrix D show the characteristic of gradually reducing detours from front to back, so as to facilitate the rapid and accurate finding of appropriate updated travel information.

[0042] When generating the adjustment plan matrix D, it is necessary to screen out some task routes that cannot be completed. The current screening method is to generate routes first and then screen them, which results in a low degree of visualization of the screening process, thereby increasing the difficulty of manual inspection. To this end, this application provides the following technical solutions:

[0043] The information base generation unit generates an adjustment scheme matrix D based on the following steps;

[0044] S1: The target region is represented by G; G = (V, E), where V represents the set of key nodes in the target region and E represents the set of edges in the target region;

[0045] S2: construct the adjacency matrix A; A = V*V;

[0046] The elements in the adjacency matrix A are Agh ; If there is an edge between node g and node h, then A gh is 1, otherwise it is 0; g≠h, g and h are the indexes of the key nodes in the target area;

[0047] S3: Construct a path matrix P, the elements in the path matrix P are P gh ;P gh represents the set of all paths from key node g to key node h;

[0048] P gh =∪ k∈V (A gk ·P kj );

[0049] S4: Find all paths from the location of the industrial robot to the target location from the path matrix P, and fill each path to generate an initial matrix R;

[0050] S5: According to the maximum driving speed and working task time limit of the industrial robot, delete the unachievable paths in the initial matrix R and calculate the direction value K i , to generate the adjustment scheme matrix D.

[0051] In the technical solution provided by the present application, by counting the number of non-placeholder elements in each row, the length of each path can be quickly calculated, and whether there are repeated paths in the route matrix can be checked. In addition, it is also possible to analyze whether there are shared nodes or edges between different paths. Therefore, the lines in the path matrix P have strong regularity, and the initial matrix R finally listed has a high degree of visualization, which is convenient for screening.

[0052] The particle swarm algorithm screens and searches for the best data in a large amount of data information through the random diffusion of particles. In order to avoid falling into the local optimal solution, the algorithm requires a large number of iterations, but this easily leads to a long algorithm calculation time, thereby reducing the response efficiency of the command. To this end, this application provides the following technical solutions:

[0053] The particle swarm update diffusion unit uses the following steps to generate the best update progress information:

[0054] Step 1: Set the fitness function, divide the adjustment scheme matrix D into several equidistant intervals, randomly generate the same number of particles in each interval, calculate the fitness value of each particle, and record the individual optimal solution and the global optimal solution;

[0055] Step 2: For each particle, update its velocity and position;

[0056] v s (t+1)=w·v s (t)+c 1 ·r1 ·(pBest s -x s (t))+c 2 ·r 2 ·(gBest-x s (t));

[0057] Among them, s represents the index of the particle, v s (t) represents the velocity of particle s at time t, w represents the inertia weight, c 1 and c 2 represents the learning factor, r 1 and r 2 is the random number introduced, pBest i represents the individual optimal solution of particle i, and gBest represents the global optimal solution;

[0058] Step 3: When the maximum number of iterations is reached or the fitness value meets the requirements, the algorithm terminates to screen out the global optimal solution.

[0059] In the technical solution proposed in this application, the adjustment scheme matrix D is divided into several equidistant intervals, and particles are randomly generated in each interval. During the iteration process, the particles will diffuse evenly from the adjustment scheme matrix D. The arrangement of the travel information in the adjustment scheme matrix D is distributed according to the adjustment value K. This regular arrangement helps to quickly determine the update direction of the particles, thereby improving the optimization efficiency.

[0060] Furthermore, c 1 >c 2 ;

[0061] In the technical solution provided in this application, c 1 It reflects the particle’s memory or recollection of its own historical experience and determines the tendency of the particle to approach its best historical position. 2 It reflects the historical experience of the group of particles in terms of cooperation and knowledge sharing, and determines the tendency of particles to approach the historical best position of the group or neighborhood. 1 >c 2 In the process of particle diffusion, more attention can be paid to the memory of its own historical experience, that is, each particle is more likely to fall into its own local optimal solution. In this way, particles in each interval diffuse in their initial interval or adjacent intervals, and then each particle searches for its own local optimal solution. All particles combined together can find the global optimal solution, thereby improving optimization efficiency.

[0062] In existing scheduling schemes, emergency tasks are usually set as the highest priority. When these emergency tasks are executed, they will affect the surrounding low-authority industrial robots, causing chaos in the entire system. To this end, this application provides the following technical solutions:

[0063] The objective function is f(s);

[0064]

[0065] Among them, T is the final completion time of updating travel information, a 1 To update the distance of the travel information, a 2 To update the number of key nodes passed in the travel information, a 3 To update the number of industrial robots that pass through key nodes in the progress signal.

[0066] To this end, the objective function designed in this application incorporates factors such as completion time, number of key nodes, affected industrial robots, and distance. During the optimization process, we will strive to find a solution that makes the above four elements as small as possible to ensure that the travel task is completed while reducing the impact on the entire system. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] The drawings constituting a part of this application are used to provide a further understanding of this application, so that other features, purposes and advantages of this application become more obvious. The drawings and descriptions of the exemplary embodiments of this application are used to explain this application and do not constitute an improper limitation on this application.

[0068] In addition, throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that the components and elements are not necessarily drawn to scale.

[0069] In the attached picture:

[0070] Figure 1 Schematic diagram of the structure of the robot operation system based on visual analysis.

[0071] Figure 2 This is a structural diagram of the image information extraction module. DETAILED DESCRIPTION

[0072] Embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present application. It should be understood that the drawings and embodiments of the present application are only for exemplary purposes and are not intended to limit the scope of protection of the present application.

[0073] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other.

[0074] The present application will be described in detail below with reference to the accompanying drawings and in combination with embodiments.

[0075] Example 1: Reference Figure 1 , the robot operation system based on visual analysis includes: a map information input module, an industrial robot, a road sign module, and a server. Among them, the map information input module is used to obtain the route information of the target area and generate a route network. In practice, the target area refers to the factory area or production workshop where the robot works automatically. For example, in some intelligent ports, all cargo transportation is completed by robots. In practice, each route is at least a two-way road to avoid collisions between robots traveling in opposite directions.

[0076] Industrial robots are used to move along routes in the target area and perform work tasks. The specific tasks performed by the industrial robots and the way they perform the tasks can be set as needed. This solution only provides a scheduling method for industrial robots. For example, the industrial robot can be an AGV robot, which can automatically move to a predetermined location to carry out operations such as cargo handling or cargo disassembly.

[0077] The road sign module is equipped with multiple radio frequency tags, which are arranged at key nodes of the route network to provide location identification information to the industrial robot. The key node is the node where at least two routes intersect. The key node actually refers to the intersection in the target area. Setting a radio frequency tag at each intersection can, on the one hand, allow the industrial robot to determine its own position in real time, and on the other hand, allow the industrial robot to slow down when approaching the intersection to avoid collision with other industrial robots.

[0078] The server is used to generate travel information according to the work tasks of each industrial robot. The travel information includes the travel route and the time interval passing through each key node. The server is used to control the work tasks of these industrial robots and schedule the industrial robots. The speed of the industrial robots during the specific travel process is controlled by themselves to improve the response efficiency of emergencies. To this end, the industrial robots are equipped with cameras to adjust the movement speed of the industrial robots in real time based on the image information collected by the cameras to avoid collisions.

[0079] In order to further avoid collisions between industrial robots, the present application provides the following technical solutions: the industrial robot includes: an execution and driving module, a first communication module, an image information extraction module, an early warning module, a second communication module, and a mobile control module.

[0080] The execution and drive module is used to execute work tasks and travel information. The execution and drive module actually refers to the manipulator, travel device and other equipment on the industrial robot. For example, a robot that needs to perform a grasping task needs to be equipped with a manipulator, a laser radar and a travel device.

[0081] The image information extraction module is connected to the camera signal to extract the video sequence transmitted by the camera and extract road condition information from it. The road condition information refers to whether there are traffic conditions in front of the industrial robot's field of vision, such as whether there are stationary objects, whether there are objects suddenly intruding on the driving path, whether there are human-shaped objects moving, etc.

[0082] The warning module is used to generate warning information based on road condition information and its own moving speed. The warning information includes its own position, its own moving speed, its own acceleration and the location information of the obstacle area. The second communication module is used to broadcast the warning information through the high-frequency band. The mobile control module is used to generate the moving speed in real time based on the road condition information, the travel information and the warning information, and the moving speed includes the moving speed and the moving direction. The warning information also includes information on whether the moving speed has changed recently.

[0083] The early warning module is a module for industrial robots to generate early warning information. During the movement, industrial robots will continuously broadcast their own position, speed and observed road conditions through high-frequency signals. Therefore, each industrial robot can receive the position, speed and surrounding road conditions of the surrounding industrial robots. In this way, each industrial robot can understand its own position and surrounding traffic conditions more clearly, so that the mobile control module can flexibly adjust the movement speed of the industrial robot according to demand. The mobile control module can reasonably select the appropriate speed to control the movement of the industrial robot when the surrounding road conditions and the movement information of each robot are known. The specific control method can be implemented using models such as Visual Language Models and Vision-Language-Action Model.

[0084] Embodiment 2: Embodiment 2 provides a further optimization solution for the server based on Embodiment 1:

[0085] The server includes: a work task input module, a work task allocation module, and a priority adjustment module.

[0086] The task input module is used to input the tasks that each industrial robot needs to perform and generate the execution time of each task. Specifically, the task type and the completion time of the task are set according to the actual situation and requirements.

[0087] The work task allocation module is used to allocate each work task to each industrial robot according to the execution time and generate corresponding travel information. When allocating work tasks, the work task allocation module gives priority to industrial robots that have no work tasks or are close to the work tasks. In the actual production environment, industrial robots are usually in a saturated state, so the work task allocation module can allocate work tasks in the order of priority.

[0088] When generating the travel information of each industrial robot, it is necessary to try to avoid the industrial robot from crossing the front, back, left, or right of a key intersection. For example, if one industrial robot passes through intersection A from left to right, and another industrial robot passes through intersection A from top to bottom, the probability of the two industrial robots colliding at intersection A will increase. By staggering the time when the two industrial robots pass through the intersection, collisions can be effectively avoided.

[0089] The priority adjustment module is used to obtain the work tasks that need to adjust the completion time limit, and generate updated travel information based on the location of the target industrial robot, and at the same time obtain the corresponding key nodes in the updated travel information, and send control instructions to the key nodes. The control instructions include the travel direction within the corresponding time interval.

[0090] After each work task is assigned to the corresponding industrial robot, if the completion progress of the task needs to be accelerated, the travel speed of the industrial robot needs to be increased and the waiting time needs to be reduced when passing through each intersection. In this case, it is inevitable to adjust the time for other industrial robots to pass through the intersection. In order to avoid communication congestion and the impact on the traffic system, this solution will adjust the traffic status of each key node on the moving path of the industrial robot after adjusting the moving speed of the target industrial robot. For example, if the adjusted industrial robot needs to pass through intersection A from left to right at 10:30, the time period of intersection A from 10:29 to 10:31 can be adjusted to only allow left and right traffic, and prohibit front and back traffic. In this way, the industrial robot can pass through the intersection quickly at 10:30 without affecting other industrial robots. For other industrial robots, the number of robots that need to pass through intersection A from front and back between 10:29 and 10:31 is small, so the impact on the entire traffic system is small. After passing intersection A, these slightly affected industrial robots only need to increase their speed to make up time.

[0091] When adjusting the priority of the work tasks performed by the industrial robot, it will inevitably affect other industrial robots in the entire system. In order to avoid system confusion, this application provides the following solutions:

[0092] The priority adjustment module includes an update task acquisition unit, an information base generation unit, and a particle swarm update diffusion unit. The update task acquisition unit is used to acquire the work tasks that need to adjust the working time limit. For example, a task that needs to be completed at 10:30 is adjusted to a task that needs to be completed at 10:40. At this time, the industrial robot needs to speed up to complete the task.

[0093] An information base generation unit generates an adjustment plan matrix D based on the work tasks that require adjustment of the work time limit; Among them, d 1 Indicates the first update progress information, d 2 Indicates the second update progress information. i represents the i-th updated travel information, i represents the index of the element in the adjustment plan matrix D. The direction value K based on the updated travel information in the adjustment plan matrix D i Arrange by size.

[0094]

[0095] Where, j represents the index of the path in the i-th updated travel information, N i represents the number of paths in the i-th updated travel information, represents the vector of the jth path in the i-th updated travel information, The vector representing the industrial robot's path from its current position to the modified final position.

[0096] The adjustment scheme matrix D is a collection of all routes that the industrial robot can complete the task. In practice, all executable routes must be screened out first, and then the routes that cannot be completed are screened out according to the maximum moving speed. For example, in this scheme, the maximum moving speed of the industrial robot is 30km / h, and some routes are too long to be completed within the specified time of the maximum moving speed, so they need to be screened out.

[0097] The direction value K is the direction of each possible route. For example, if the work task is to go from point A to point B, if a certain route tries to move in this direction, then the direction value K is high. If a certain route is circling in place, then the direction value K is small. The direction value in this solution is a measure of whether each route has a detour.

[0098] The information base generation unit generates the adjustment scheme matrix D based on the following steps:

[0099] S1: The target region is represented by G; G = (V, E), where V represents the set of key nodes in the target region and E represents the set of edges in the target region;

[0100] S2: construct the adjacency matrix A; A = V*V;

[0101] The elements in the adjacency matrix A are A gh ; If there is an edge between node g and node h, then A gh is 1, otherwise it is 0; g≠h, g and h are the indexes of the key nodes in the target area;

[0102] S3: Construct a path matrix P, the elements in the path matrix P are P gh ;P gh represents the set of all paths from key node g to key node h;

[0103] P gh =∪ k∈V (A gk ·P kj );

[0104] S4: Find all paths from the location of the industrial robot to the target location from the path matrix P, and fill each path to generate an initial matrix R;

[0105] The initial matrix R refers to all possible path combinations.

[0106] S5: According to the maximum driving speed and working task time limit of the industrial robot, delete the unachievable paths in the initial matrix R and calculate the direction value K i , to generate the adjustment scheme matrix D.

[0107] The particle swarm update diffusion unit is used to generate a number of particles in the adjustment scheme matrix D and perform iterative updates to find the best updated new information.

[0108] The adjustment scheme matrix D is a combination matrix of all possible updated travel information. Which scheme in this combination matrix is ​​the best needs to be screened. Since there are too many schemes in the adjustment scheme matrix D, a particle swarm is used to update the diffusion unit for diffusion. The specific scheme is as follows:

[0109] The particle swarm update diffusion unit uses the following steps to generate the best update progress information:

[0110] Step 1: Set the fitness function, divide the adjustment scheme matrix D into several equidistant intervals, randomly generate the same number of particles in each interval, calculate the fitness value of each particle, and record the individual optimal solution and the global optimal solution.

[0111] In this scheme, the key to using the particle swarm algorithm is to divide the adjustment scheme matrix D into several equidistant intervals. The scheme matrix D in this scheme is actually the arrangement of all possible routes, which is a number axis after expansion. So it can be divided into several intervals.

[0112] The objective function is f(s).

[0113] Among them, T is the final completion time of updating travel information, a 1 To update the distance of the travel information, a 2 To update the number of key nodes passed in the travel information, a 3 To update the number of industrial robots that pass through key nodes in the progress signal, it is necessary to find the maximum value of the objective function during the update.

[0114] Step 2: For each particle, update its velocity and position;

[0115] v s (t+1)=w·v s (t)+c 1 ·r 1 ·(pBest s -x s (t))+c 2 ·r 2 ·(gBest-x s (t));

[0116] Among them, s represents the index of the particle, v s (t) represents the velocity of particle s at time t, w represents the inertia weight, c 1 and c 2 represents the learning factor, r 1 and r 2 is the introduced random number, pBest i represents the individual optimal solution of particle i, gBest represents the global optimal solution, c 1 >c 2 ;c 1 It reflects the particle’s memory or recollection of its own historical experience and determines the tendency of the particle to approach its best historical position. 2 It reflects the historical experience of the group, which is the collaborative cooperation and knowledge sharing among particles, and determines the tendency of particles to approach the historical best position of the group or neighborhood.

[0117] Step 3: When the maximum number of iterations is reached or the fitness value meets the requirements, the algorithm terminates to screen out the global optimal solution.

[0118] Therefore, in this scheme, by controlling c 1 >c 2 , and make the initial positions of the particles evenly distributed in the adjustment scheme matrix D, so that the optimal solution can be found quickly.

[0119] refer to Figure 2, Embodiment 3: Embodiment 3 provides an image information extraction module on the basis of Embodiment 1. The extraction of road condition information in this solution is a necessary condition to ensure the normal operation of industrial robots. The features extracted by current image information extraction technologies are mainly based on static objects, and it is impossible to accurately find the required information from video sequences.

[0120] For this reason, in this embodiment, the image information extraction module includes: an input layer, an enhancement layer, a processing layer, and an output layer.

[0121] Among them, the input layer includes n×n cells, and n is consistent with the resolution of the input image frame. The membrane potential excitation value of the cell is the brightness change value at adjacent moments. I(x, y, t) = L(x, y, t) - L(x, y, t - 1), where x and y are the abscissa and ordinate of the cell, and t is the time index.

[0122] In order to reduce the input of image hardening and increase the ability to capture motion information, it is necessary to output uncertain excitation information P(x, y, t).

[0123]

[0124] Among them, α represents the enhancement coefficient of the contrast between the moving target and the background, and ψ n represents the signal noise threshold, which is used to filter out tiny clutter signals in the field of view, and G 1 (i, j) is a filtering mask.

[0125] ε i is a constant used to adjust the amplitude or intensity of the Gaussian function, and σ i is the standard deviation of the Gaussian function, which determines the width or spread of the Gaussian distribution.

[0126] The enhancement layer includes an ON information extraction layer and an OFF information extraction layer. The ON information extraction layer receives the inflow into the ON channel, that is, it only receives the output of the cells in the PHC layer whose excitation value is greater than zero. In the ON information extraction layer, the visual excitation output L on (x, y, t) at position (x, y) is determined by the following formula:

[0127]

[0128] [x] + represents the max(0, x) operation to achieve half-wave rectification. Z(x, y, t) is defined as G 2 (x, y)·H(t), where G 2 (x, y) represents a filtering mask used to implement the lateral inhibition mechanism, and H(t) is a function that fits the excitation residue coefficient.

[0129]

[0130] In the formula, and T are adaptive parameters.

[0131] The OFF information extraction layer receives visual signals flowing into the OFF channel, that is, it only receives the output of cells in the PHC layer whose excitation values ​​are less than zero. off The output L of the cell at position (x,y) in the sublayer of (x,y,t) is determined by:

[0132] [x] - Indicates the min(0,x) operation, which is used to implement half-wave rectification. off (x, y, t) represents the output after the visual signal is enhanced by the excitation residual mechanism and the lateral inhibition mechanism.

[0133] The input layer is responsible for extracting basic brightness change information from the original input image. It draws on the ON (On-center) and OFF (Off-center) channel characteristics of the biological visual system to respond to changes in brightness increase and decrease in the image respectively.

[0134] The ON channel produces an excitatory response to areas where the local brightness of the image is too high, while the OFF channel responds to areas where the local darkness is too low, thus providing the basic dual-channel features for subsequent processing.

[0135] The enhancement layer receives dual-channel input data from the brightness change detection module, and its input tensor shape is (H, W, 2), where H and W represent the height and width of the image respectively.

[0136] The enhancement layer processes and enhances the signals of ON and OFF channels independently to highlight important visual features and suppress noise.

[0137] Processing layer: Receives the dual-channel enhanced feature map and uses a 5x5 convolution kernel to extract spatial information. By stacking multiple convolution layers with ReLU activation functions, a deep convolutional network structure is constructed to effectively capture a wider range of motion information and complex features in the image. Finally, a feature tensor with a shape of (H, W, N) is output, where N represents the number of channels of the final convolution layer.

[0138] The output layer inputs the multi-channel feature map extracted by the convolutional network into a specially designed target detection module. By applying an appropriate loss function (such as cross entropy or mean square error) in this output layer, the training model accurately identifies the moving objects in the image. These marked moving areas are the main focus of road condition information, thus helping the autonomous driving system make corresponding decisions.

[0139] The above description is only some preferred embodiments of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present application is not limited to the technical solutions formed by the specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above invention concept. For example, the above features are replaced with (but not limited to) the technical features with similar functions disclosed in the embodiments of the present application to form a technical solution.

Claims

1. A robot operation system based on visual analysis, characterized in that: include: A map information input module is used to obtain route information of the target area and generate a route network; Industrial robots, which are used to move along routes and perform work tasks within a target area; The road sign module is equipped with multiple radio frequency tags, which are arranged at key nodes of the route network to provide location identification information to the industrial robot; A server, used for generating travel information according to the work tasks of each industrial robot; The travel information includes the travel route and the time interval passing through each key node; Among them, the industrial robot is equipped with a camera for adjusting the movement speed of the industrial robot in real time based on image information collected by the camera.

2. The robot operation system based on visual analysis according to claim 1, characterized in that: The key node is a node where at least two routes intersect.

3. The robot operation system based on visual analysis according to claim 2, characterized in that: Industrial robots include: Execution and driving module, used to execute work tasks and travel information; The first communication module is used to establish a signal connection with the server to obtain work tasks and travel information; An image information extraction module is connected to the camera signal and is used to extract the video sequence transmitted by the camera and extract road condition information from it; An early warning module is used to generate early warning information based on road condition information and its own moving speed, wherein the early warning information includes its own position, its own moving speed, its own acceleration and obstacle area position information; The second communication module is used to broadcast the warning information externally through the high frequency band; The mobile control module is used to generate a mobile speed in real time based on road condition information, travel information and warning information, wherein the mobile speed includes a moving rate and a moving direction.

4. The robot operation system based on visual analysis according to claim 3, characterized in that: The warning information also includes information on whether the moving speed has changed recently.

5. The robot operation system based on visual analysis according to any one of claims 1 to 4, characterized in that: The server includes: A work task input module is used to input the work tasks that need to be performed by each industrial robot and generate the execution time of each work task; The work task allocation module allocates each work task to each industrial robot according to the execution time and generates corresponding travel information; The priority adjustment module obtains the work tasks whose completion time limit needs to be adjusted, generates updated travel information based on the location of the target industrial robot; obtains the corresponding key nodes in the updated travel information, and sends control instructions to the key nodes; the control instructions include the travel direction within the corresponding time interval.

6. The robot operation system based on visual analysis according to claim 5, characterized in that: The priority adjustment module includes: Update the task acquisition unit to acquire the work tasks that need to adjust the work time limit; An information base generating unit generates an adjustment plan matrix D based on the work tasks that require adjustment of the work time limit; Wherein, d1 represents the first updated traveling information, and d2 represents the second updated traveling information; i represents the i-th updated traveling information, i represents the index of the element in the adjustment scheme matrix D; the direction value K based on the updated traveling information in the adjustment scheme matrix D i Arrange by size; Where, j represents the index of the path in the i-th updated travel information, N i represents the number of paths in the i-th updated travel information, represents the vector of the jth path in the i-th updated travel information, The vector representing the industrial robot from its current position to the modified final position; The particle swarm update diffusion unit is used to generate a number of particles in the adjustment scheme matrix D and perform iterative updates to find the best updated new information.

7. The robot operation system based on visual analysis according to claim 6, characterized in that: The information base generation unit generates an adjustment scheme matrix D based on the following steps; S1: The target region is represented by G; G = (V, E), where V represents the set of key nodes in the target region and E represents the set of edges in the target region; S2: construct the adjacency matrix A; A = V*V; The elements in the adjacency matrix A are A gh ; If there is an edge between node g and node h, then A gh is 1, otherwise it is 0; g≠h, g and h are the indexes of the key nodes in the target area; S3: Construct a path matrix P, the elements in the path matrix P are P gh ;P gh represents the set of all paths from key node g to key node h; P gh =U k∈V (A gk ·P kj ); S4: Find all paths from the location of the industrial robot to the target location from the path matrix P, and fill each path to generate an initial matrix R; S5: According to the maximum driving speed and working task time limit of the industrial robot, delete the unachievable paths in the initial matrix R and calculate the direction value K i , to generate the adjustment scheme matrix D.

8. The robot operation system based on visual analysis according to claim 6, characterized in that: Step 1: Set the fitness function, divide the adjustment scheme matrix D into several equidistant intervals, randomly generate the same number of particles in each interval, calculate the fitness value of each particle, and record the individual optimal solution and the global optimal solution; Step 2: For each particle update its velocity and position: v s (t+1)=w·v s (t)+c1·r1·(pBest s -x s (t))+c2·r2·(gBest- x s (t)); Among them, s represents the index of the particle, v s (t) represents the velocity of particle s at time t, w represents the inertia weight, c1 and c2 represent the learning factors, r1 and r2 are the introduced random numbers, pBest i represents the individual optimal solution of particle i, and gBest represents the global optimal solution; Step 3: When the maximum number of iterations is reached or the fitness value meets the requirements, the algorithm terminates to screen out the global optimal solution.

9. The robot operation system based on visual analysis according to claim 8, characterized in that: c1>c2.

10. The robot operation system based on visual analysis according to claim 8, characterized in that: The objective function is f(s);

Images