Industrial vision robot interaction method and system

Abstract

The present invention relates to the technical field of robot interaction, and specifically discloses an industrial vision robot interaction method and system. The industrial vision robot interaction method comprises: acquiring scene video information, scene point cloud data, and natural language instruction information; acquiring a plurality of real-time scene images on the basis of the scene video information, and extracting features from each real-time scene image to obtain real-time scene pixel point features; acquiring a plurality of historical scene images on the basis of scene cloud data image information, and extracting features from each historical scene image to obtain historical scene pixel point features; and acquiring scene difference feature points distinguishing the real-time scene pixel point features from the historical scene pixel point features. In the present invention, by capturing scene video information and extracting pixel point features of real-time scene images in real time, dynamic changes of a scene can be comprehensively sensed in real time. The real-time responsiveness of the system to the environment is ensured, so that the system can adapt to various dynamic environments.

Description

Industrial vision robot interaction method and system TECHNICAL FIELD

[0001] The present application relates to the technical field of robot interaction, in particular to an industrial vision robot interaction method and system. BACKGROUND

[0002] In the industrial field, industrial vision robots have been widely used in automobile manufacturing, electronic equipment, food packaging, pharmaceutical production and other industries. For example, in automobile manufacturing, industrial vision robots can be used for detection and assembly of parts; in electronic equipment production, it can detect circuit boards and control the quality of welding points; in food packaging, it can detect the appearance and packaging quality of food.

[0003] Existing vision robot systems often show inadequate adaptability when facing unstructured scenes. They usually can only operate according to pre-set navigation paths and tasks, lacking real-time perception and response ability to environmental changes, which means that in complex or dynamic industrial environments, robots may not be able to effectively respond to unexpected situations, thereby affecting the safety and efficiency of work.

[0004] SUMMARY

[0005] The purpose of the present application is to provide an industrial vision robot interaction method to solve the technical problems raised in the background.

[0006] To achieve the above purpose, the present application provides the following technical scheme:

[0007] The industrial vision robot interaction method comprises:

[0008] Obtaining scene video information, scene point cloud data and natural language instruction information;

[0009] According to the scene video information, a plurality of real-time scene images are obtained, and features are extracted from each real-time scene image to obtain real-time scene pixel point features;

[0010] According to the scene cloud data image information, a plurality of historical scene images are obtained, and features are extracted from each historical scene image to obtain historical scene pixel point features;

[0011] Obtaining scene difference feature points that distinguish real-time scene pixel point features and historical scene pixel point features;

[0012] According to the natural language instruction information, voice execution information and target scene information are obtained,

[0013] Based on the scene difference feature points and the target scene information, a three-dimensional scene is created to obtain three-dimensional semantic scene information;

[0014] The interactive information is acquired according to the voice execution information and the three-dimensional semantic scene information, and path planning and action execution are performed based on the interactive information.

[0015] Preferably, the step of acquiring a plurality of real-time scene images according to the scene video information and extracting features of each real-time scene image to obtain real-time scene pixel point features comprises:

[0016] A plurality of scene image frames are acquired according to the scene video information.

[0017] The plurality of scene image frames are read frame by frame in a loop, and in each iteration of the loop, a plurality of scene frames are captured, and the scene frames are taken as real-time scene images.

[0018] The real-time scene images are sequentially subjected to grayscale, noise reduction and contrast enhancement preprocessing to obtain enhanced scene images.

[0019] Features of the enhanced scene images are extracted to obtain real-time scene pixel point features.

[0020] Preferably, the step of acquiring a plurality of historical scene images according to the scene cloud data image information and extracting features of each historical scene image to obtain historical scene pixel point features comprises:

[0021] Scene cloud data image information is acquired based on scene cloud data.

[0022] The historical scene images are acquired by eliminating noise, adjusting size and performing color space conversion preprocessing according to the scene cloud data image information.

[0023] Pixel features of each historical scene image are extracted to obtain scene pixel point features.

[0024] The scene pixel point features are normalized to obtain historical scene pixel point features.

[0025] Preferably, the step of acquiring scene difference feature points that are different from the real-time scene pixel point features and the historical scene pixel point features comprises:

[0026] A first scene pixel feature vector is acquired according to the real-time scene pixel point features.

[0027] A second scene pixel feature vector is acquired according to the historical scene pixel point features.

[0028] A feature point matching vector value is obtained by cosine similarity calculation according to the first scene pixel feature vector and the second scene pixel feature vector.

[0029] It is judged whether the feature point matching vector value is in a preset interval.

[0030] If yes, judging whether the real-time scene pixel point feature and the historical scene pixel point feature are similar;

[0031] If no, judging whether the real-time scene pixel point feature and the historical scene pixel point feature are similar, and taking the dissimilar real-time scene pixel point feature as a scene difference feature point.

[0032] Preferably, the step of obtaining the voice execution information and the target scene information according to the natural language instruction information comprises:

[0033] Obtaining the natural language instruction information, and performing text extraction on the natural language instruction information to obtain text language information;

[0034] Performing word segmentation processing on the text language information to obtain a plurality of semantic words;

[0035] Performing part-of-speech tagging on each semantic word to obtain tagged words;

[0036] Obtaining semantic entities corresponding to the tagged words based on a preset vocabulary table;

[0037] Obtaining a grammatical structure of the semantic entities, and splicing the plurality of semantic entities based on the grammatical structure to obtain a plurality of spliced semantic entities;

[0038] Performing semantic extraction on the spliced semantic entities based on an intent recognition model to obtain semantic understanding information and scene semantic information;

[0039] Obtaining the voice execution information according to the semantic understanding information;

[0040] Obtaining the target scene information according to the scene semantic information.

[0041] Preferably, the step of creating a three-dimensional scene based on the scene difference feature point and the target scene information to obtain three-dimensional semantic scene information comprises:

[0042] Obtaining a difference pixel value according to the scene difference feature point;

[0043] Obtaining a depth value of the difference pixel value;

[0044] Obtaining a target voxel according to the target scene information;

[0045] Obtaining a depth value of the target voxel value;

[0046] Calculating a voxel space value according to the difference pixel and the target voxel, wherein the space value calculation formula is: sdf(V xyz )=D(L ij )-C(V xyz );

[0047] wherein, sdf(V xyz ) is the space value, D(Lij ) is a depth value of the difference pixel L ij , L ij is a depth value of the difference pixel, C(V xyz ) is a depth value of the target voxel value, V xyz is the target voxel value;

[0048] According to the spatial value, the signed distance value of the voxel is calculated, and the calculation formula is:

[0049] Wherein, tsdf(V xyz ) is the signed distance value of the voxel, and t is the stage distance value of the tsdf.

[0050] Obtain the semantic information of the target voxel;

[0051] According to the semantic information and the signed distance value, the spatial center coordinates of each target scene information are obtained by averaging;

[0052] Based on the spatial center coordinates of each target scene information, a three-dimensional scene is created, and three-dimensional semantic scene information is obtained.

[0053] Preferably, the step of obtaining interaction information according to the voice execution information and the three-dimensional semantic scene information, and performing path planning and action execution based on the interaction information, comprises:

[0054] According to the voice execution information, the intention information and the target information are obtained;

[0055] According to the target information, the target position information in the three-dimensional semantic scene information is obtained;

[0056] According to the combination of the intention information and the target position information, the interaction information is obtained, wherein the interaction information comprises target position, expected action information and obstacle position in the environment;

[0057] Based on the interaction information, the path information from the current position to the target position is obtained;

[0058] Based on the expected action information, the control instruction information is generated;

[0059] According to the control instruction information, the action is executed according to the path information.

[0060] The application also provides an industrial visual robot interactive control system, comprising:

[0061] A first acquisition module is used for acquiring scene video information, scene point cloud data and natural language instruction information;

[0062] A second acquisition module is used for acquiring a plurality of real-time scene images according to the scene video information, and extracting features of each real-time scene image to obtain real-time scene pixel features.

[0063] a third obtaining module, configured to obtain a plurality of historical scene images according to scene cloud data image information, and extract features of each historical scene image to obtain historical scene pixel point features;

[0064] a fourth obtaining module, configured to obtain scene difference feature points that are different between real-time scene pixel point features and historical scene pixel point features;

[0065] a fifth obtaining module, configured to obtain voice execution information and target scene information according to natural language instruction information,

[0066] a creating module, configured to create a three-dimensional scene based on the scene difference feature points and the target scene information to obtain three-dimensional semantic scene information;

[0067] an interactive execution module, configured to obtain interactive information according to the voice execution information and the three-dimensional semantic scene information, and perform path planning and action execution based on the interactive information.

[0068] The application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the industrial visual robot interaction method when executing the computer program.

[0069] The application further provides a computer readable storage medium, which stores a computer program, and the computer program implements the steps of the industrial visual robot interaction method when executed by a processor.

[0070] The application has the advantages that the application can comprehensively and in real time perceive the dynamic changes of the scene by capturing the scene video information and extracting the pixel point features of the real-time scene images in real time. This ensures the real-time response capability of the system to the environment, enabling it to adapt to various dynamic environments; not only focusing on the real-time scene, but also realizing the combination of history and reality by obtaining historical scene images and extracting their features. This combination helps the system to more comprehensively understand the scene, identify the changes in the scene by comparing real-time and historical data, and make more accurate decisions. By comparing the real-time scene pixel point features and the historical scene pixel point features, the difference feature points in the scene can be accurately identified. These difference feature points may be important changes or events, which are crucial for decision-making and response, can analyze natural language instructions, obtain voice execution information and target scene information, can understand and execute human natural language instructions, thereby realizing the intelligentization of human-computer interaction, based on the scene difference feature points and the target scene information, accurate three-dimensional semantic scene information can be created. This three-dimensional scene information not only contains the geometric information of the scene, but also contains the semantic information, which can more deeply understand the scene, according to the voice execution information and the three-dimensional semantic scene information, the interaction information can be obtained, and efficient path planning and action execution can be carried out based on these information. This ensures that the system can quickly and accurately respond to various instructions and tasks. BRIEF DESCRIPTION OF DRAWINGS

[0071] Fig. 1 is a method flowchart of the application.

[0072] Fig. 2 is a schematic diagram of the system structure of the application.

[0073] The implementation, functional characteristics and advantages of the application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0074] It should be understood that the specific embodiments described herein are only used to explain the application and not to limit the application.

[0075] As shown in Fig. 1, the application provides an industrial vision robot interaction method, comprising:

[0076] S1, obtaining scene video information, scene point cloud data and natural language instruction information;

[0077] S2, obtaining a plurality of real-time scene images according to the scene video information, extracting features for each real-time scene image to obtain real-time scene pixel point features;

[0078] S3, obtaining a plurality of historical scene images according to the scene cloud data image information, extracting features for each historical scene image to obtain historical scene pixel point features;

[0079] S4, acquire scene difference feature points distinguishing real-time scene pixel point features and historical scene pixel point features;

[0080] S5, acquire voice execution information and target scene information according to natural language instruction information,

[0081] S6, create three-dimensional scene based on scene difference feature points and target scene information, and obtain three-dimensional semantic scene information;

[0082] S7, acquire interaction information according to voice execution information and three-dimensional semantic scene information, and perform path planning and action execution based on the interaction information.

[0083] As described in steps S1-S7 above, there may be various complex factors such as light changes, occlusions, shadows, and reflections in industrial environments, which may affect the recognition and positioning of target objects by the vision system. The adaptability of robots to unstructured environments is limited, and it is difficult to handle unexpected situations or dynamically changing environments. Currently, industrial vision robots mostly interact through pre-set programs or instructions, lacking flexibility and intelligence. Robots are difficult to understand and respond to complex human instructions or requirements, and cannot interact naturally and efficiently with humans.

[0084] The application can obtain real-time and dynamic scene content through video information, and point cloud data provides depth information and geometric structure of the scene, and natural language instruction information provides clear operation guidance, solves the problem that the traditional method may only rely on single visual information, leading to inaccurate understanding of the scene in some cases (such as light change, occlusion, etc.), a plurality of real-time scene images are obtained according to scene video information, features of each real-time scene image are extracted, key information of the image can be obtained through feature extraction, which is helpful for subsequent scene understanding and recognition, direct processing of original video data may lead to large amount of calculation and slow processing speed, feature extraction can significantly reduce the amount of calculation and improve the processing speed, a plurality of historical scene images are obtained according to scene cloud data image information, historical data provide long-term and stable understanding of the scene, which is helpful for making up for the deficiency of real-time data, by comparing the features of real-time scene and historical scene, changes in the scene can be identified, and the scene is further understood, scene difference feature points that are different from real-time scene pixel point features and historical scene pixel point features are obtained, by comparing the features of real-time scene and historical scene, changes in the scene can be detected, such as newly added objects, moving objects, etc., the changes in the scene that cannot be effectively detected by the traditional method can be detected, and the problem that the understanding of the scene is inaccurate can be solved, through natural language instructions, the user can interact with the system, and clear operation guidance is provided, the obtained voice execution information and target scene information provide clear task target and execution mode for the system, three-dimensional scene creation is carried out based on the scene difference feature points and the target scene information, the obtained three-dimensional semantic scene information not only contains geometric information of the scene, but also contains semantic information such as the category and position of the object, based on the rich three-dimensional semantic information and the clear voice execution information, intelligent decision and planning can be made, and the user-specified task can be efficiently completed through path planning and action execution.

[0085] In one embodiment, the step of obtaining a plurality of real-time scene images according to the scene video information, and extracting features of each real-time scene image to obtain real-time scene pixel point features comprises:

[0086] S201, obtaining a plurality of scene image frames according to scene video information;

[0087] S202, reading the plurality of scene image frames frame by frame in a loop, and in each iteration of the loop, capturing a plurality of scene frames, and taking the scene frames as real-time scene images;

[0088] S203, sequentially performing grayscale, noise reduction and contrast enhancement preprocessing on the real-time scene images to obtain enhanced scene images;

[0089] S204, performing feature extraction on the enhanced scene images to obtain real-time scene pixel point features.

[0090] As described in steps S201-S204, the application obtains multiple scene image frames through scene video information, can capture continuous image frames from the video stream in real time, provides a basis for subsequent image processing and analysis, through continuous capture of multiple image frames, can completely reflect the dynamic changes of the scene, then reads each frame in the loop, and in each iteration of the loop, captures multiple scene frames, then through frame-by-frame reading and capturing, can continuously process each frame of image, ensures the coherence of processing, then sequentially performs grayscale, noise reduction, contrast enhancement preprocessing on the real-time scene image, reduces the data amount of the image, speeds up the subsequent processing speed, while retaining the contour information of the image; noise reduction: removes noise in the image, improves the definition and quality of the image; contrast enhancement: enhances the contrast of the image, makes the target object in the image more prominent, facilitates subsequent recognition and analysis, through feature extraction, can extract key information and features from the image, for subsequent recognition, classification and other tasks.

[0091] In one embodiment, the step of obtaining multiple historical scene images according to scene cloud data image information, and extracting features from each historical scene image to obtain historical scene pixel point features, comprises:

[0092] S301, obtaining scene cloud data image information based on scene cloud data;

[0093] S302, performing noise elimination, size adjustment, and color space conversion preprocessing according to the scene cloud data image information to obtain multiple historical scene images;

[0094] S303, performing pixel feature extraction on each historical scene image to obtain scene pixel point features;

[0095] S304, normalizing the scene pixel point features to obtain historical scene pixel point features.

[0096] As described in steps S301-S304, the application can obtain a large amount of historical scene image information through scene cloud data, which provides a rich data source for subsequent analysis and processing. The pre-processing steps such as noise removal, size adjustment, and color space conversion can effectively improve the quality of the image and reduce the error of subsequent processing. Then, pixel feature extraction is performed on each historical scene image to quickly and accurately obtain the key information in the image, providing a basis for subsequent applications. Finally, the scene pixel feature is normalized to ensure that the features between different images are consistent and comparable, thereby improving the accuracy of subsequent analysis. Through the pre-processing steps, the problems of noise, inconsistent size, and color space mismatch in the original image can be effectively solved, improving the quality of the image. It is often difficult to extract useful feature information directly from the original image, while the pixel-based feature extraction method can quickly and accurately obtain the key information in the image. Due to differences in shooting conditions, equipment, and other factors between different images, the features between images may not be consistent. By normalizing the scene pixel feature, the inconsistency can be eliminated, improving the accuracy of subsequent analysis.

[0097] In one embodiment, the step of obtaining scene difference feature points that distinguish the real-time scene pixel point feature and the historical scene pixel point feature comprises:

[0098] S401, obtaining a first scene pixel feature vector according to the real-time scene pixel point feature;

[0099] S402, obtaining a second scene pixel feature vector according to the historical scene pixel point feature;

[0100] S403, obtaining a feature point matching vector value through cosine similarity calculation based on the first scene pixel feature vector and the second scene pixel feature vector;

[0101] S404, determining whether the feature point matching vector value is within a preset interval;

[0102] If it is within, it is determined that the real-time scene pixel point feature and the historical scene pixel point feature are similar;

[0103] If it is not within, it is determined that the real-time scene pixel point feature and the historical scene pixel point feature are not similar, and the real-time scene pixel point feature that is not similar is taken as a scene difference feature point.

[0104] As described in steps S401-S404 above, the present application can accurately determine the similarity between the real-time scene pixel point features and the historical scene pixel point features by calculating the cosine similarity of the two. This method takes advantage of the directionality of the feature vector and can capture subtle changes in the scene. Cosine similarity calculation is an efficient feature matching method that can quickly compare the similarity between two feature vectors, thereby determining whether the real-time scene and the historical scene are similar. The preset interval can be adjusted according to actual needs to achieve different levels of difference detection. For example, the interval can be adjusted to detect minor scene changes or significant scene changes. By automatically identifying and marking scene difference feature points, it can more intelligently understand and respond to environmental changes, providing strong support for subsequent decision-making and action, and solving the problem of low efficiency and susceptibility to subjective factors in existing manual scene difference detection methods. By automatically calculating feature point matching vector values, these limitations can be overcome, improving the accuracy and efficiency of detection, and traditional image processing methods may be sensitive to changes in lighting and perspective, leading to false positives. By extracting pixel point features and calculating the similarity between feature vectors, this sensitivity can be reduced.

[0105] In one embodiment, the step of obtaining voice execution information and target scene information according to natural language instruction information comprises:

[0106] S501, obtaining natural language instruction information and performing text extraction on the natural language instruction information to obtain text language information;

[0107] S502, performing word segmentation processing on the text language information to obtain a plurality of semantic words;

[0108] S503, performing part-of-speech tagging on each semantic word to obtain tagged words;

[0109] S504, obtaining semantic entities corresponding to the tagged words based on a preset vocabulary;

[0110] S505, obtaining the grammatical structure of the semantic entities and concatenating the plurality of semantic entities based on the grammatical structure to obtain a plurality of concatenated semantic entities;

[0111] S506, performing semantic extraction on the concatenated semantic entities based on an intent recognition model to obtain semantic understanding information and scene semantic information;

[0112] S507, obtaining voice execution information according to the semantic understanding information;

[0113] S508, obtaining target scene information according to the scene semantic information.

[0114] As described in steps S501-S508, the application can interact with the system in a more natural and intuitive way through natural language instruction information, without the need to learn specific commands or syntax, improving user experience, while being able to understand and analyze natural language instructions from users, extract semantic understanding information and scene semantic information, and then perform corresponding operations or provide related services, demonstrating the high intelligence of the system. Through steps such as word segmentation, part-of-speech tagging, and semantic entity extraction of text language information, the system can quickly and accurately identify the user's intent, reducing processing time and errors. Based on the preset vocabulary and intent recognition model, the system can handle natural language instructions in different fields and scenarios, with strong adaptability. Finally, by obtaining target scene information, the system can apply in multiple scenarios according to the context information of the user's instructions, such as smart home, intelligent customer service, and autonomous driving.

[0115] In one embodiment, the step of creating a three-dimensional scene based on the scene difference feature points and the target scene information to obtain three-dimensional semantic scene information comprises:

[0116] S601, obtaining a difference pixel value according to the scene difference feature points;

[0117] S602, obtaining a depth value of the difference pixel value;

[0118] S603, obtaining a target voxel according to the target scene information;

[0119] S604, obtaining a depth value of the target voxel value;

[0120] S605, calculating a voxel space value according to the difference pixel and the target voxel, wherein the space value calculation formula is: xyz )=D(L ij )-C(V xyz );

[0121] Wherein, sdf(V xyz ) is the space value, D(L ij ) is the depth value of the difference pixel L ij , L ij is the difference pixel, C(V xyz ) is the depth value of the target voxel value, and V xyz is the target voxel value.

[0122] S606, calculating a signed distance value of the voxel according to the space value, wherein the calculation formula is:

[0123] Wherein, tsdf(V xyz ) is the signed distance value of the voxel, and t is the stage distance value of tsdf.

[0124] S607, obtaining semantic information of the target voxel;

[0125] S608, obtaining spatial center coordinates of each target scene information by averaging according to the semantic information and the signed distance value;

[0126] S609, creating a three-dimensional scene based on the spatial center coordinates of each target scene information to obtain three-dimensional semantic scene information.

[0127] As described in the above steps S601-S609, by obtaining the difference pixel value and its depth value, as well as the target voxel and its depth value, the voxel space value and the signed distance value can be accurately calculated, thereby obtaining accurate three-dimensional spatial information. This helps to create more accurate and more realistic three-dimensional semantic scene information, and then based on the semantic information and the signed distance value, the spatial center coordinates of each target scene information are obtained by averaging. This automatic calculation reduces human intervention and errors, improves the efficiency and accuracy of three-dimensional scene creation, and then by obtaining the scene difference feature points and their pixel values, changes in the scene can be quickly identified, and subsequent three-dimensional scene creation is performed accordingly. This helps to quickly respond to environmental changes and meets the needs of real-time applications. Then, in the process of creating a three-dimensional scene, not only the depth information of pixels and voxels is considered, but also the semantic information of the target voxel is integrated, which makes the created three-dimensional semantic scene information not only have accurate spatial structure, but also contain rich semantic content, which is helpful for subsequent scene understanding, analysis and application. The three-dimensional semantic scene information created thereafter can be widely used in virtual reality, augmented reality, intelligent monitoring, robot navigation and other fields, which can provide intuitive and realistic three-dimensional scene information for these fields and promote the development and application of related technologies.

[0128] In one embodiment, the step of obtaining interaction information according to the voice execution information and the three-dimensional semantic scene information, and performing path planning and action execution based on the interaction information, comprises:

[0129] S701, obtaining intention information and target information according to the voice execution information;

[0130] S702, obtaining target position information in the three-dimensional semantic scene information according to the target information;

[0131] S703, combining the intention information and the target position information to obtain interaction information, wherein the interaction information includes target position, expected action information and obstacle position in the environment;

[0132] S704, obtaining path information from the current position to the target position based on the interaction information;

[0133] S705, generating control instruction information based on the expected action information.

[0134] As described in steps S701-S705, the present application parses the intent information and target information in the voice execution information, enabling the system to more accurately understand the user's intent and locate the target position in the three-dimensional semantic scene information. This interactive mode is more natural and intuitive, greatly enhancing the user experience, and providing path information from the current position to the target position for the user or robot based on the interaction information, including the target position, expected action information, and obstacle position in the environment. This intelligent navigation and planning function ensures efficient and safe path selection, avoiding potential obstacles and conflicts, and based on the expected action information, the system can generate corresponding control instruction information, thereby realizing real-time control of the robot or other devices. This rapid response and action execution capability enables the system to quickly complete the user's intent, improving the system's execution efficiency. Since the system is based on voice execution information and three-dimensional semantic scene information for interaction and control, it has strong flexibility and scalability. Users can control the system through simple voice instructions without complex programming or operation. At the same time, the system can easily adapt to different environmental and task requirements, expanding its application range by updating the three-dimensional semantic scene information, and avoiding potential safety hazards and conflicts by combining the target position, expected action information, and obstacle position in the interaction information. This intelligent obstacle avoidance and path planning function ensures the safety and reliability of the system's operation, reducing the risk of accidents and failures.

[0135] As shown in FIG. 2, the present application also provides a control system for an industrial visual robot interaction method, comprising:

[0136] A first acquisition module 1 is configured to acquire scene video information, scene point cloud data, and natural language instruction information;

[0137] A second acquisition module 2 is configured to acquire a plurality of real-time scene images based on the scene video information, extract features from each real-time scene image, and obtain real-time scene pixel point features;

[0138] A third acquisition module 3 is configured to acquire a plurality of historical scene images based on the scene cloud data image information, extract features from each historical scene image, and obtain historical scene pixel point features;

[0139] A fourth acquisition module 4 is configured to acquire scene difference feature points that distinguish the real-time scene pixel point features and the historical scene pixel point features;

[0140] A fifth acquisition module 5 is configured to acquire voice execution information and target scene information based on the natural language instruction information,

[0141] The creating module 6 is configured to create a three-dimensional scene based on the scene difference feature points and the target scene information, and obtain three-dimensional semantic scene information.

[0142] The interactive executing module 7 is configured to acquire interactive information according to the voice executing information and the three-dimensional semantic scene information, and perform path planning and action execution based on the interactive information.

[0143] The application further provides a computer device, including a memory and a processor, the memory stores a computer program, and the processor implements the steps of the industrial visual robot interaction method when executing the computer program.

[0144] The application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps of the industrial visual robot interaction method.

[0145] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing related hardware, and the computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. Any reference to memory, storage, database or other medium provided by the application and used in the embodiments can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM) and memory bus dynamic RAM (RDRAM).

[0146] It is to be understood that the terminology "including", "comprising", or any other variation thereof, is intended to cover a non-exclusive inclusion such that process, methods, articles, or apparatuses that comprise a list of elements do not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without limitation, an element preceded by "comprises a" does not, without more constraints, foreclose the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0147] The above description is merely the preferred embodiments of the present application, and is not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made according to the content of the present application specification and drawings, or directly or indirectly applied to other related technical fields, is also included in the patent protection scope of the present application.

Claims

1. An industrial vision robot interaction method, characterized in that, The method comprises the steps of: acquiring scene video information, scene point cloud data and natural language instruction information; acquiring a plurality of real-time scene images according to the scene video information, extracting features of each real-time scene image, and obtaining real-time scene pixel point features; acquiring a plurality of historical scene images according to the scene cloud data image information, extracting features of each historical scene image, and obtaining historical scene pixel point features; acquiring scene difference feature points that are different from the real-time scene pixel point features and the historical scene pixel point features; acquiring voice execution information and target scene information according to the natural language instruction information, creating a three-dimensional scene based on the scene difference feature points and the target scene information, and obtaining three-dimensional semantic scene information; acquiring interaction information according to the voice execution information and the three-dimensional semantic scene information, and performing path planning and action execution based on the interaction information.

2. The industrial visual robotic interaction method of claim 1, wherein, The step of acquiring a plurality of real-time scene images according to the scene video information, extracting features of each real-time scene image, and obtaining real-time scene pixel point features comprises the steps of: acquiring a plurality of scene image frames according to the scene video information; frame-by-frame reading in a loop is performed on the plurality of scene image frames, and in each iteration of the loop, a plurality of scene frames are captured, and the scene frames are taken as real-time scene images; graying, noise reduction and contrast enhancement preprocessing are sequentially performed on the real-time scene images to obtain enhanced scene images; feature extraction is performed on the enhanced scene images to obtain real-time scene pixel point features.

3. The industrial visual robotic interaction method of claim 1, wherein, The step of acquiring a plurality of historical scene images according to the scene cloud data image information, extracting features of each historical scene image, and obtaining historical scene pixel point features comprises the steps of: acquiring scene cloud data image information based on scene cloud data; performing noise elimination, size adjustment and color space conversion preprocessing on the scene cloud data image information to obtain a plurality of historical scene images; pixel feature extraction is performed on each historical scene image to obtain scene pixel point features; normalization is performed on the scene pixel point features to obtain historical scene pixel point features.

4. The industrial vision robot interaction method of claim 1, wherein, The step of acquiring scene difference feature points that are different from the real-time scene pixel point features and the historical scene pixel point features comprises the steps of: acquiring a first scene pixel feature vector according to the real-time scene pixel point features; acquiring a second scene pixel feature vector according to the historical scene pixel point features; obtaining a feature point matching vector value through cosine similarity calculation according to the first scene pixel feature vector and the second scene pixel feature vector; determining whether the feature point matching vector value is in a preset interval; if yes, determining that the real-time scene pixel point features and the historical scene pixel point features are similar; if no, determining that the real-time scene pixel point features and the historical scene pixel point features are not similar, and taking the real-time scene pixel point features that are not similar as scene difference feature points.

5. The industrial vision robot interaction method of claim 1, wherein, The step of acquiring voice execution information and target scene information according to the natural language instruction information comprises the steps of: acquiring natural language instruction information, and performing text extraction on the natural language instruction information to obtain text language information; performing word segmentation processing on the text language information to obtain a plurality of semantic words; performing part-of-speech tagging on each semantic word to obtain tagged words; acquiring semantic entities corresponding to the tagged words based on a preset vocabulary table; The syntax structure of the semantic entity is acquired, and a plurality of semantic entities are spliced based on the syntax structure to obtain a plurality of spliced semantic entities; The spliced semantic entities are subjected to semantic extraction based on an intent recognition model to obtain semantic understanding information and scene semantic information; Voice execution information is acquired according to the semantic understanding information; Target scene information is acquired according to the scene semantic information.

6. The industrial vision robot interaction method of claim 5, wherein, The step of creating a three-dimensional scene based on the scene difference feature points and the target scene information to obtain three-dimensional semantic scene information comprises: Difference pixel values are acquired according to the scene difference feature points; Depth values of the difference pixel values are acquired; Target voxels are acquired according to the target scene information; Depth values of the target voxel values are acquired; A voxel space value is calculated according to the difference pixel and the target voxel, wherein the space value calculation formula is: sdf(V xyz ) = D(L ij )-C(V xyz ); wherein sdf(V xyz ) is a spatial value, D(L ij ) is a depth value of a difference pixel L ij , L ij is a difference pixel, C(V xyz ) is a depth value of a target voxel value, and V xyz is a target voxel value. The signed distance value of the voxel is calculated according to the spatial value, and the calculation formula is: wherein tsdf(V xyz ) is the signed distance value of the voxel, and t is the stage distance value of the tsdf. Semantic information of the target voxel is acquired; A space center coordinate of each target scene information is obtained by averaging the semantic information and the symbolic distance value; A three-dimensional scene is created based on the space center coordinate of each target scene information to obtain three-dimensional semantic scene information.

7. The industrial visual robotic interaction method of claim 1, wherein, The step of acquiring interaction information according to the voice execution information and the three-dimensional semantic scene information, and performing path planning and action execution based on the interaction information comprises: Intent information and target information are acquired according to the voice execution information; Target position information in the three-dimensional semantic scene information is acquired according to the target information; Interaction information is obtained by combining the intent information and the target position information, wherein the interaction information comprises target position, expected action information and obstacle position in the environment; Path information from a current position to the target position is acquired based on the interaction information; Control instruction information is generated based on the expected action information; Action execution is performed according to the control instruction information and the path information.

8. An industrial vision robot interaction control system, characterized by, It comprises: A first acquisition module is configured to acquire scene video information, scene point cloud data and natural language instruction information; A second acquisition module is configured to acquire a plurality of real-time scene images from the scene video information, and extract features from each real-time scene image to obtain real-time scene pixel point features; A third acquisition module is configured to acquire a plurality of historical scene images from the scene cloud data image information, and extract features from each historical scene image to obtain historical scene pixel point features; A fourth acquisition module is configured to acquire scene difference feature points that are different between the real-time scene pixel point features and the historical scene pixel point features; A fifth acquisition module is configured to acquire voice execution information and target scene information from the natural language instruction information, A creation module is configured to create a three-dimensional scene based on the scene difference feature points and the target scene information to obtain three-dimensional semantic scene information; An interaction execution module is configured to acquire interaction information according to the voice execution information and the three-dimensional semantic scene information, and perform path planning and action execution based on the interaction information. 9.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-8 when the computer program is executed by the processor. The processor executes the computer program to implement the steps of the industrial visual robot interaction method in any one of claims 1 to 7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the industrial visual robot interaction method in any one of claims 1 to 7.

Images