Virtual soft measurement method and system for machine vision training based on digital twinning

Abstract

Embodiments of the present application provide a virtual soft measurement method and system for machine vision training based on digital twinning. The method comprises: obtaining an image with target object edge features from an image database by a vision twin main core, and dynamically updating a virtual model of an industrial environment in a model database; the vision twin main core collects customized training data for an image processing model in the dynamically updated virtual model according to customized demand configuration information, and sends the customized training data to an artificial intelligence training end; and the embedded end re-mounts the image processing model trained by the artificial intelligence training end, to realize dynamic virtual soft measurement of the industrial environment. Embodiments of the present application can generate usable training data close to a real scene in a virtual model according to different industrial environment needs, improve the efficiency and reliability of machine vision construction, and the embedded end introduced in the method also reduces the data processing and data transmission burden of the vision twin main core.

Description

Technical Field

[0001] This invention relates to the field of machine vision, and more particularly to a virtual software testing method and system for machine vision training based on digital twins. Background Technology

[0002] With the advancement of technology, the application of machine vision has expanded to industrial environments in many fields, including agricultural production, metal processing, chemical monitoring, and pharmaceutical manufacturing. Due to the new demands for industrial automation and intelligence, the importance of machine vision is increasing daily, and its performance and reliability are continuously improving.

[0003] Data acquisition and AI training are two fundamental components of building a machine vision system. In existing machine vision deployment paradigms, these two parts are heavily coupled: training data must be collected first and then sent to the training device, and the trained AI is then deployed in the camera. Therefore, data acquisition and AI training cannot be performed simultaneously, inevitably impacting the efficiency of machine vision system deployment.

[0004] Data acquisition is often time-consuming and labor-intensive. Take product quality inspection as an example: machine vision systems used for quality inspection require a large number of images of defective products as training data. However, with current quality engineering technology, defective products are often extremely rare, making it difficult to obtain a sufficient number of defective product images. Therefore, collecting defective product data often requires extensive product production, inevitably impacting the setup of the machine vision system. If the environment or target changes, retraining is necessary, further affecting the efficiency of machine vision system setup. In existing technologies, data augmentation is commonly used to address the difficulty of obtaining training data. Data augmentation refers to partially modifying real image data, such as occlusion, color gamut alteration, or flipping, to achieve a significant increase in the amount of training data. The augmented training set then undergoes subsequent processing such as parameter tuning and model adaptation to optimize the final training effect.

[0005] In the process of realizing this invention, the inventors discovered at least the following problems in the related technology:

[0006] Because the new data generated by data augmentation is not based on real-world conditions, the training effect on machine vision is often unstable. The augmented training set, due to its consistent scene representation, frequently leads to overfitting after training, meaning the accuracy of machine vision is only reliable in specific scenarios. Therefore, data augmentation still has a significant limitation in improving the efficiency of machine vision system deployment.

[0007] Even though existing technologies utilize virtual software testing tools to address machine vision training issues, these tools provide analysis, simulation, and verification capabilities in various scenarios by creating a virtual simulation environment. In current technologies, virtual software testing tools effectively simplify the development process because testing hardware is often unavailable or difficult to set up. In the field of machine vision, virtual software testing tools can quickly and specifically expand training data through simulation, effectively addressing the problem of overfitting. However, due to limited data volume, current virtual software testing tools still struggle to realistically reproduce mechanisms in the physical world. Furthermore, virtual software testing tools lack a direct connection to real-world scenarios, and their updates rely on manual operation and are often not periodic, inevitably leading to data lag in the virtual model. Moreover, the construction of virtual software testing tools is usually based on engineers' knowledge, experience, and understanding, which inevitably results in deviations from actual conditions. Summary of the Invention

[0008] To address at least the problems of overfitting in machine vision training and the lack of direct correlation between virtual software testing and reality in existing technologies, this invention provides a virtual software testing method for machine vision training based on digital twins, comprising:

[0009] The image processing model on the embedded device is used to preprocess the dynamically captured images in the industrial environment, and the preprocessed image with the edge features of the target object is input into the image database.

[0010] The image with the edge features of the target object is obtained from the image database through the visual twin master kernel, and the virtual model of the industrial environment in the model database is dynamically updated.

[0011] The visual twin master core collects customized training data for the image processing model in the dynamically updated virtual model according to the customized requirements configuration information, and sends the customized training data to the artificial intelligence training terminal to simultaneously collect the customized training data and train the image processing model.

[0012] The embedded terminal is reloaded with the image processing model trained by the artificial intelligence training terminal to achieve automated control training of machine vision in the industrial environment.

[0013] Secondly, embodiments of the present invention provide a virtual software testing system for machine vision training based on digital twins, comprising:

[0014] The image processing module is used to preprocess images dynamically captured in an industrial environment using the image processing model on the embedded terminal, and input the preprocessed image with the edge features of the target object into the image database.

[0015] The virtual model update program module is used to obtain the image with the edge features of the target object from the image database through the visual twin master kernel, and dynamically update the virtual model of the industrial environment in the model database.

[0016] The training module is used by the visual twin master core to collect customized training data for the image processing model in the dynamically updated virtual model according to the customized requirements configuration information, and send the customized training data to the artificial intelligence training terminal to simultaneously collect the customized training data and train the image processing model.

[0017] The dynamic software testing module is used to reload the image processing model trained by the artificial intelligence training terminal on the embedded terminal, so as to realize automated control training of machine vision in the industrial environment.

[0018] Thirdly, an electronic device is provided, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of a virtual software testing method for machine vision training based on digital twins according to any embodiment of the present invention.

[0019] Fourthly, embodiments of the present invention provide a storage medium storing a computer program thereon, characterized in that the program is executed by a processor to implement the steps of a virtual software testing method for machine vision training based on digital twins according to any embodiment of the present invention.

[0020] The beneficial effects of this invention are as follows: In virtual software testing, the virtual model is dynamically updated according to the actual environment, ensuring the timeliness of the data and ensuring that usable training data close to the real scene can be generated in the virtual model according to the needs of different industrial environments, thereby improving the efficiency and reliability of machine vision construction. At the same time, the embedded terminal introduced in the method also reduces the data processing and data transmission burden of the visual twin master core. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1This is a flowchart of a virtual software testing method for machine vision training based on digital twins, provided by an embodiment of the present invention.

[0023] Figure 2 This is a schematic diagram of digital twin collaboration for a virtual software testing method for machine vision training based on digital twin, provided by an embodiment of the present invention.

[0024] Figure 3 This is a schematic diagram of a visual twin framework for a virtual software testing method for machine vision training based on digital twins, provided in an embodiment of the present invention.

[0025] Figure 4 This is a schematic diagram of the visual twin operation process of a virtual software testing method for machine vision training based on digital twins, provided by an embodiment of the present invention.

[0026] Figure 5 This is a schematic diagram of a visual twin modularization method for virtual software testing of machine vision training based on digital twin, provided in an embodiment of the present invention.

[0027] Figure 6 This is a schematic diagram of a working scenario for a virtual software testing method for machine vision training based on digital twins, provided in an embodiment of the present invention.

[0028] Figure 7 This is a schematic diagram of a virtual generated scene image provided by an embodiment of the present invention for a virtual software testing method for machine vision training based on digital twins.

[0029] Figure 8 This is a schematic diagram of tree outline drawing for a virtual software testing method for machine vision training based on digital twins, provided in an embodiment of the present invention.

[0030] Figure 9 This is a schematic diagram of an edge recognition model for a virtual software testing method for machine vision training based on digital twins, provided in an embodiment of the present invention.

[0031] Figure 10 This is a schematic diagram of the training model parameter settings for a virtual software testing method for machine vision training based on digital twins, provided in an embodiment of the present invention.

[0032] Figure 11 This is a schematic diagram of the contour drawing of a real scene for a virtual software testing method for machine vision training based on digital twins, provided in an embodiment of the present invention.

[0033] Figure 12 This is a graph showing the height and position data of a tree in a scene, provided by an embodiment of the present invention, which is a virtual software testing method for machine vision training based on digital twins.

[0034] Figure 13 This is a schematic diagram illustrating the automatic construction of a virtual model for a scenario of a virtual software testing method for machine vision training based on digital twins, provided by an embodiment of the present invention.

[0035] Figure 14 This is a schematic diagram of the structure of a virtual software testing system for machine vision training based on digital twins, provided in an embodiment of the present invention.

[0036] Figure 15 This is a schematic diagram of an embodiment of an electronic device for virtual software testing of machine vision training based on digital twins, provided as an embodiment of the present invention. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0038] like Figure 1 The diagram shows a flowchart of a virtual software testing method for machine vision training based on digital twins, according to an embodiment of the present invention, including the following steps:

[0039] S11: Use the image processing model on the embedded terminal to preprocess the dynamically captured images in the industrial environment, and input the preprocessed image with the edge features of the target object into the image database.

[0040] S12: Obtain the image with the edge features of the target object from the image database through the visual twin master kernel, and dynamically update the virtual model of the industrial environment in the model database;

[0041] S13: The visual twin master core collects customized training data for the image processing model in the dynamically updated virtual model according to the customized requirements configuration information, and sends the customized training data to the artificial intelligence training terminal to simultaneously collect the customized training data and train the image processing model.

[0042] S14: The embedded terminal reloads the image processing model trained by the artificial intelligence training terminal to realize automated control training of machine vision in the industrial environment.

[0043] Digital twins are the core methodology of visual twins. A digital twin is a high-precision, multi-physical, multi-dimensional, probabilistic simulation model built on data and models to reflect the state of a physical object. It represents the concept of real-time, high-precision mapping of the physical environment and the instantaneous state of the target object into a virtual space. A digital twin consists of three main parts: a physical space, a virtual space, and data interaction between the two spaces. Digital twins have been researched and implemented in many key environments of industrial manufacturing, such as conceptual design, intelligent customization, maintenance management, and supply chain management. Figure 2 As shown, digital twins can be a key component in solving machine vision training problems. The efficiency of machine vision is limited by data acquisition and its coupling with training. Virtual software testing tools provide a platform for simultaneous virtual data generation, acquisition, and training. Digital twins further improve accuracy, data timeliness, and reliability based on virtual software testing tools. Digital twins can be integrated with machine vision, but how to acquire large amounts of training data and how to use the training data for machine time training are still problems that need to be solved. Therefore, this method uses digital twins to drive the simultaneous data generation and acquisition-training of machine vision, which can be called visual twins. Figure 3 The framework for visual twins in this method includes an industrial environment, a vision system, an embedded terminal, a visual twin master core, user interaction, and an artificial intelligence training terminal.

[0044] In this embodiment, for step S11, machine vision can be applied to industrial environments in many fields such as agricultural and forestry monitoring, metal processing, chemical monitoring, and pharmaceutical manufacturing, and typically has different uses in different industrial environments. The industrial environment includes: a monitoring environment for the target object and a motion control environment for the target object; specifically, monitoring the state and characteristics of the target object in a certain environment, for example, an industrial environment monitoring the growth progress of agricultural and forestry plants; or the motion control state of an object, such as the motion control state of a robotic arm in an automated production line environment.

[0045] Taking the monitoring environment of target objects as an example, this method can be applied to industrial environments for monitoring the growth progress of agricultural and forestry plants. These environments typically contain trees, rice paddies, rivers, utility poles running through forests, and other target objects. This method can also be applied to industrial environments for product quality inspection in metal processing, which typically contain various parts as target objects. Machine vision used for agricultural and forestry monitoring focuses on the spacing, height, and size of plants. Machine vision used for product quality inspection, on the other hand, focuses on product category, defect parameters, and lighting effects. In other words, there is no universal model for the visual twin generated by this method; it must be defined according to the user's needs and the specific application scenario, and will be customized for different scenarios and applications.

[0046] Vision systems represent monitoring cameras in industrial environments. Taking forestry growth monitoring as an example, in some areas, high-voltage power towers are built in forests. However, as trees grow, they may touch the power lines, creating safety hazards. Monitoring cameras can typically be installed on objects near the forest area, such as on high-voltage power towers. This ensures sufficient visibility for the cameras and reduces interference from surrounding environmental factors. The main monitoring targets of these cameras include grass, trees, and high-voltage lines. The cameras dynamically capture image information containing both the monitored targets and environmental information within their field of view.

[0047] The embedded terminal represents the edge computing platform connected to the monitoring camera. It serves as a bridge between the machine vision system and the visual twin master core. The embedded terminal performs the following tasks: 1. Preprocessing images transmitted from the monitoring camera, such as compression, conversion, and filtering; 2. Equipped with a trained machine vision AI model for image processing, which supports the embedded system's automatic processing and vision application functions; 3. Also a communication platform, capable of directly sending captured image information, image preprocessing results, and machine vision task execution results to other servers or embedded terminals, while simultaneously receiving data from other servers or embedded terminals.

[0048] When the industrial environment is the monitoring environment of the target object, the preprocessing includes: cleaning and / or image compression and / or color gamut conversion and / or edge detection;

[0049] When the industrial environment is a motion control environment for the target object, the preprocessing includes: image segmentation and / or edge detection.

[0050] In this embodiment, because "visual twin" is a system method built for visual information acquisition and processing in a wide range of industrial scenarios, it needs to be customized according to specific industrial environments and visual information problems. Due to the synchronization and high-precision modeling of physical and virtual areas, a large number of data processing and transmission tasks are inevitable. If all of these are handled by the visual twin main core, the burden on the main core would be too heavy. To further reduce the computational load on the visual twin main core, an image processing model on the embedded device is used to preprocess the images, thus sharing the data processing and data transmission burden of the visual twin.

[0051] Different preprocessing techniques are applied to different industrial environments. The preprocessed image, containing the edge features of the target object, is then input into an image database. In general, the embedded system performs compression, conversion, filtering, and edge detection on the initial image information dynamically captured by the monitoring camera. An edge is defined as a sudden change in grayscale or structural information. Examples include changes in grayscale levels, colors, and texture structures. An edge marks the end of one region and the beginning of another; this feature can be used to segment an image. This ensures the visual twin kernel can directly model the image information, reducing its workload. Specifically, in a visual twin built for product quality inspection machine vision, image preprocessing on the embedded system requires cleaning, compression, color gamut conversion, and edge detection for each image. For machine vision used for robotic arm motion control, the preprocessing task on the embedded system focuses more on image segmentation and edge detection. The above are just examples of some scenarios and are not intended to limit the scope.

[0052] For step S12, the visual twin master core of this method refers to a virtual platform that maps machine vision in real time. The master core is built on a cloud server or a local server. It has the following functions: real-time monitoring of the scene in machine vision and training customized virtual models for machine vision (used to simulate virtual scenes that reflect real-world scenarios). As the monitoring camera continuously captures images of the industrial environment (e.g., target trees are constantly growing taller), the embedded terminal continuously expands the image database. The visual twin master core obtains the latest images of the target trees from the image database and dynamically updates the virtual model of the industrial environment to ensure its timeliness.

[0053] In step S13, the visual twin master kernel also generates customized training data for the image processing model (wherein, the customized training data includes the virtually generated image of the corresponding scene and its edge feature image), and also undertakes the task allocation of data processing and data transmission. Figure 4 As shown, visual twins operate along two main functions: virtual mapping of machine vision and customization of training data.

[0054] Customization of training data refers to the targeted generation of training data from the established virtual model. Based on the customized configuration information, the main core retrieves a virtual scene from the database and randomly generates images from different perspectives within the scene according to data requirements. After further processing, the generated images are sent to the AI ​​training terminal for simultaneous acquisition of customized training data and training of the image processing model. The customized configuration information is input by the user through the visual twin's interactive interface. In this embodiment, the user can configure various information in the interactive interface: 1. The user can manually create and modify the virtual model in the visual twin; 2. The user can control and adjust the generation of virtual training data; 3. The user can manage and set the visual twin's database. User intervention can effectively improve the efficiency and quality of the visual twin, thereby customizing the user's configuration information.

[0055] In step S14, after the AI ​​training end trains the image processing model using customized training data, it sends the trained image processing model to the embedded end, which then reloads the trained model. In this way, the embedded end continues to provide the visual twin master core with real-time images of target object edge features in an industrial environment using the trained image processing model. The image processing model in this method is not trained using existing real images; instead, it uses images of the real-world scene as a mapping to establish a WYSIWYG virtual model that mirrors the real-world scene. Within this virtual model, images that meet the user's customized requirements are generated.

[0056] The above steps illustrate a virtual software testing process where an image processing model is configured on an embedded device. However, initially, the embedded device does not have an image processing model and needs to train an initial image processing model before proceeding. Figure 4 As shown.

[0057] As one implementation, before mounting the image processing model on the embedded device, the method includes:

[0058] The visual twin master core performs image post-processing on training images of an industrial environment input from an embedded terminal, wherein the image post-processing includes: cleaning and / or image segmentation and / or color gamut conversion and / or image compression and / or edge detection;

[0059] The object types and features of each object in the post-processed training image are identified, the target object is determined, and the training image is compiled into a spatial image of a three-dimensional point cloud.

[0060] The visual twin master kernel performs virtual mapping of the target object in the spatial image of the three-dimensional point cloud using machine vision, constructs a virtual model of the industrial environment, and stores the virtual model in the model database.

[0061] The visual twin master core collects customized training data in the virtual model according to the customized requirements and configuration information, and sends the customized training data to the artificial intelligence training terminal for training the image processing model.

[0062] The AI ​​training terminal loads the trained image processing model onto the embedded terminal.

[0063] In this embodiment, images captured by a monitoring camera within an industrial environment can be used as training images. These training images are then input into the visual twin master core via an embedded terminal. The visual twin master core performs various post-processing operations on the training images, including image cleaning, image segmentation, color gamut conversion, image compression, and edge detection, to more clearly identify target objects in the training images and compile the training images into a spatial image of a three-dimensional point cloud.

[0064] As one implementation method, the spatial information of the industrial environment can be determined by compiling the image data type of the target object in the spatial image of the three-dimensional point cloud using a visual twin master core based on a real-time rendering engine.

[0065] A virtual model of the industrial environment is constructed based on the spatial information.

[0066] In this embodiment, the visual twin platform software is developed using a real-time rendering engine, although most existing digital twin applications choose 3D solid modeling software as the development platform. 3D solid modeling software is renowned for its comprehensiveness and realism in multi-physical quantity simulation and is widely used in digital twin research on aircraft, CNC machine tools, and wind turbine blades. However, in this method, visual information is the primary data type, while physical quantity characteristics such as length, volume, and amount of matter are not emphasized. Therefore, multi-physical quantity simulation in 3D solid modeling software would become a burden on the computational resources of this method. Some companies develop specialized digital twin software for specific scenarios and applications, such as Plant Simulation and Azure for freight transportation. These digital twin software programs, due to specific optimizations, can achieve good simulation results while moderately consuming computational resources. However, these specialized software programs are developed for specific scenarios, have limited applicability, and lack precision in image information. In contrast, real-time rendering engines are specifically developed for image processing and editing, achieving excellent image realism without wasting computational resources, making them the most suitable for this method. After constructing a virtual model of the industrial environment, it is stored in a model database for later retrieval / update.

[0067] Similarly, with a virtual model, customized training data can be collected in the virtual model according to the configuration information based on customized needs, and the customized training data can be sent to the artificial intelligence training terminal to train the image processing model. The resulting image processing model will be mounted on the embedded terminal to reduce the data processing burden of the visual twin main core in subsequent virtual software testing.

[0068] In summary, the visual twin structure of this method is as follows: Figure 5 As shown, the visual twin is divided into seven parts: an embedded end, a monitoring end, a main server, an interactive interface, a modeling end, a storage repository, and a simulation end. The embedded end handles image preprocessing and carries trained artificial intelligence for image processing. The monitoring end reflects the scene in machine vision in real time. The main server is the central hub for data processing and transmission. The interactive interface is the user's window to the visual twin. The modeling end handles automatic model generation and customization. The storage repository manages and stores all relevant data and models. The simulation end generates virtual data.

[0069] As can be seen from this implementation method, the virtual model is dynamically updated according to the actual environment in virtual software testing, ensuring the timeliness of the data and ensuring that usable training data close to the real scene can be generated in the virtual model according to the needs of different industrial environments. This improves the efficiency and reliability of machine vision construction. At the same time, the embedded terminal introduced in the method also reduces the data processing and data transmission burden of the visual twin master core.

[0070] This method is illustrated with specific experiments, and preliminary attempts have been made in practice. For example... Figure 6 As shown in the two scenarios, this method was used for the protection of a high-voltage transmission tower in a certain area. Many of the high-voltage transmission towers in this area are located in woodland. To prevent trees from growing and touching the power lines, it is necessary to continuously monitor tree height to send early warnings. Due to the area's lack of development and poor accessibility, manual monitoring is very difficult (meaning that obtaining realistic images from multiple angles is relatively difficult in this scenario; even using surveillance cameras, only images from a few angles can be obtained, making training data acquisition extremely challenging). Therefore, after consultation with relevant departments, the early warning system will be automated using cameras installed on the transmission towers. Due to time constraints and poor accessibility, it is currently difficult to collect sufficient scene images to train the AI ​​deployed in the cameras for image processing. Therefore, it was decided to apply this method, visual twins, to achieve virtual software testing for training and monitoring.

[0071] The deployed camera is a Sony IMX219 with a resolution of 3648*2736 and a focal length of 3.04mm. The embedded system is powered by a Raspberry Pi 4 with 8GB of cache and a 1.5GHz quad-core processor. The training host is equipped with a 3.7GHz processor. The system features a W-2255 processor, an Nvidia RTX 3070 graphics card with 32GB of cache, and Visual Twin's software platform is Blender 2.93.

[0072] Initially, only 20 scene images were collected on-site. The plan involves first constructing an automatic virtual model using a visual twin reference to the scene photos, and then generating customized virtual images based on the virtual model. These virtual images (i.e., customized training data) are used to train an image processing model (artificial intelligence) deployed on the camera's embedded device. After training, the image processing model actively captures images of target objects (e.g., trees) in the scene and performs real-time modeling to achieve the monitoring effect. The artificial intelligence model uses the most commonly used convolutional neural network. Figure 7 As shown, the training data is generated by randomly generating particle effects in Blender. These randomized objects include terrain, grass particle effects, tree particle effects, and the sizes of trees and grass. Furthermore, since the colors of trees and grass in the scene are all green and relatively constant, to prevent the convolutional neural network from using the green color gamut as a recognition standard rather than color difference, the scene's color gamut is also randomly generated within a set range. Additionally, the scheme infers tree height by measuring the tree's outline, such as... Figure 8 As shown, the training data also includes the outlines of trees generated corresponding to the scenes, which are used as training data sets. The tree outline generation is implemented using the OpenCV library in Python. Ultimately, a total of 400 sets of scene and outline images are generated as the training set. To improve training performance, the scheme selects 10 real images and outline images as parameter tuning data for the convolutional neural network.

[0073] The convolutional neural network model specifically employs an edge recognition method. A specific model is shown below. Figure 9 As shown, the image first undergoes noise reduction, then edge detection is performed using each pixel, and finally rendering is applied to improve edge image quality. Specific training parameter settings are as follows: Figure 10 As shown. During training, 400 sets of image data were segmented into 22,000 image patches, of which 1 / 7 were edge image patches. 10 real images were segmented into 5,500 image patches for training and correction, of which 1 / 7 were edge image patches.

[0074] The trained model will then be used to monitor scene conditions and reflect them through automated modeling. Since the scene's terrain and background are largely constant, they are pre-set. The key points to determine are the tree's position and height. Because the tower's geometric parameters, camera parameters, and tower spacing are known, the tree's position and true height can be automatically derived using a set of trigonometric functions, given the overall outline of the tree. To validate the solution, such as... Figure 11As shown, two real images are randomly selected and fed into a trained neural network model, which then outputs a drawn contour. Because the overall color difference of the tree is not significant and its color is similar to that of the grass, the output contour image contains a large amount of noise. Therefore, the outer contour of the output image is extracted and noise is further removed to obtain the geometric information of the contour. Through geometric derivation, the corresponding position and height of the tree are obtained as follows: Figure 12 As shown. Based on the obtained position and height information, the scene is automatically constructed as follows. Figure 13 As shown, virtual software testing of this industrial environment was implemented.

[0075] like Figure 14 The diagram shown is a schematic diagram of a virtual software testing system for machine vision training based on digital twins according to an embodiment of the present invention. The system can execute the virtual software testing method for machine vision training based on digital twins as described in any of the above embodiments and is configured in a terminal.

[0076] This embodiment provides a virtual software testing system 10 for machine vision training based on digital twins, which includes: an image processing program module 11, a virtual model update program module 12, a training program module 13, and a dynamic software testing program module 14.

[0077] The image processing module 11 is used to preprocess images dynamically captured in an industrial environment using an image processing model mounted on the embedded terminal, and input the preprocessed image with the edge features of the target object into the image database; the virtual model update module 12 is used to obtain the image with the edge features of the target object from the image database through the visual twin master core, and dynamically update the virtual model of the industrial environment in the model database; the training module 13 is used by the visual twin master core to collect customized training data for the image processing model in the dynamically updated virtual model according to the customized requirements configuration information, and send the customized training data to the artificial intelligence training terminal, and simultaneously collect the customized training data and train the image processing model; the dynamic software testing module 14 is used to reload the image processing model trained by the artificial intelligence training terminal on the embedded terminal, so as to realize the automated control training of machine vision in the industrial environment.

[0078] This invention also provides a non-volatile computer storage medium storing computer-executable instructions that can execute the virtual software testing method for machine vision training based on digital twins in any of the above method embodiments.

[0079] In one embodiment, the non-volatile computer storage medium of the present invention stores computer-executable instructions, which are configured as follows:

[0080] The image processing model on the embedded device is used to preprocess the dynamically captured images in the industrial environment, and the preprocessed image with the edge features of the target object is input into the image database.

[0081] The image with the edge features of the target object is obtained from the image database through the visual twin master kernel, and the virtual model of the industrial environment in the model database is dynamically updated.

[0082] The visual twin master core collects customized training data for the image processing model in the dynamically updated virtual model according to the customized requirements configuration information, and sends the customized training data to the artificial intelligence training terminal to simultaneously collect the customized training data and train the image processing model.

[0083] The embedded terminal is reloaded with the image processing model trained by the artificial intelligence training terminal to achieve automated control training of machine vision in the industrial environment.

[0084] As a non-volatile computer-readable storage medium, it can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the methods in the embodiments of the present invention. One or more program instructions are stored in the non-volatile computer-readable storage medium, and when executed by a processor, they perform the virtual software testing method for machine vision training based on digital twins in any of the above method embodiments.

[0085] Figure 15 This is a schematic diagram of the hardware structure of an electronic device for a virtual software testing method for machine vision training based on digital twins, as provided in another embodiment of this application. Figure 15 As shown, the device includes:

[0086] One or more processors 1510 and memory 1520, Figure 15 Taking a processor 1510 as an example, the device for a virtual software testing method for machine vision training based on digital twins may also include an input device 1530 and an output device 1540.

[0087] The processor 1510, memory 1520, input device 1530, and output device 1540 can be connected via a bus or other means. Figure 15 Taking the example of a connection between China and Israel via a bus.

[0088] The memory 1520, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the virtual software testing method for machine vision training based on digital twins in the embodiments of this application. The processor 1510 executes various server functions and data processing by running the non-volatile software programs, instructions, and modules stored in the memory 1520, thereby implementing the virtual software testing method for machine vision training based on digital twins as described in the above embodiments.

[0089] The memory 1520 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and applications required for at least one function; the data storage area may store data, etc. Furthermore, the memory 1520 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 1520 may optionally include memory remotely located relative to the processor 1510, and these remote memories may be connected to the mobile device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0090] Input device 1530 can receive input numerical or character information. Output device 1540 may include display devices such as a display screen.

[0091] The one or more modules are stored in the memory 1520, and when executed by the one or more processors 1510, they execute the virtual software testing method for machine vision training based on digital twins in any of the above method embodiments.

[0092] The above-described product can perform the methods provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects for performing the methods. Technical details not described in detail in this embodiment can be found in the methods provided in the embodiments of this application.

[0093] Non-volatile computer-readable storage media may include a stored program area and a stored data area, wherein the stored program area may store an operating system and an application program required for at least one function; the stored data area may store data created based on the use of the device, etc. Furthermore, the non-volatile computer-readable storage medium may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the non-volatile computer-readable storage medium may optionally include memory remotely located relative to the processor, and these remote memories may be connected to the device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0094] This invention also provides an electronic device comprising: at least one processor and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of the virtual software testing method for machine vision training based on digital twins according to any embodiment of this invention.

[0095] The electronic devices described in this application exist in various forms, including but not limited to:

[0096] (1) Mobile communication devices: These devices are characterized by their mobile communication capabilities and primarily aim to provide voice and data communication. These terminals include smartphones, multimedia phones, feature phones, and low-end phones.

[0097] (2) Ultra-mobile personal computer devices: These devices fall under the category of personal computers, possessing computing and processing capabilities, and generally also have mobile internet access features. These terminals include PDAs, MIDs, and UMPCs, such as tablet computers.

[0098] (3) Portable entertainment devices: These devices can display and play multimedia content. This category includes audio and video players, handheld game consoles, e-book readers, as well as smart toys and portable car navigation devices.

[0099] (4) Other electronic devices with data processing functions.

[0100] In this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, without necessarily requiring or implying any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising" or "including" include not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0101] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0102] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0103] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A virtual software testing method for machine vision training based on digital twins, comprising: The image processing model on the embedded device is used to preprocess the dynamically captured images in the industrial environment, and the preprocessed image with the edge features of the target object is input into the image database. The image with the edge features of the target object is obtained from the image database through the visual twin master kernel, and the virtual model of the industrial environment in the model database is dynamically updated. The visual twin master core collects customized training data for the image processing model in the dynamically updated virtual model according to the customized requirements configuration information, and sends the customized training data to the artificial intelligence training terminal to simultaneously collect the customized training data and train the image processing model. The embedded terminal is reloaded with the image processing model trained by the artificial intelligence training terminal to achieve automated control training of machine vision in the industrial environment.

2. The method according to claim 1, wherein, Before the image processing model is mounted on the embedded device, the method includes: The visual twin master core performs image post-processing on training images of an industrial environment input from an embedded terminal, wherein the image post-processing includes: cleaning and / or image segmentation and / or color gamut conversion and / or image compression and / or edge detection; The object types and features of each object in the post-processed training image are identified, the target object is determined, and the training image is compiled into a spatial image of a three-dimensional point cloud. The visual twin master kernel performs virtual mapping of the target object in the spatial image of the three-dimensional point cloud using machine vision, constructs a virtual model of the industrial environment, and stores the virtual model in the model database. The visual twin master core collects customized training data in the virtual model according to the customized requirements and configuration information, and sends the customized training data to the artificial intelligence training terminal for training the image processing model. The AI ​​training terminal loads the trained image processing model onto the embedded terminal.

3. The method according to claim 2, wherein, The virtual mapping of the target object to the spatial image of the 3D point cloud using the visual twin master kernel includes: The spatial information of the industrial environment is determined by compiling the image data type of the target object in the spatial image of the three-dimensional point cloud using a visual twin kernel based on a real-time rendering engine. A virtual model of the industrial environment is constructed based on the spatial information.

4. The method according to claim 1, wherein, The industrial environment includes: the monitoring environment of the target object and the motion control environment of the target object; When the industrial environment is a monitoring environment for the target object, the preprocessing includes: cleaning and / or image compression and / or color gamut conversion and / or edge detection; When the industrial environment is a motion control environment for the target object, the preprocessing includes: image segmentation and / or edge detection.

5. The method according to claim 1, wherein, The customized configuration information is input by the user through the interactive interface of the visual twin kernel.

6. A virtual software testing system for machine vision training based on digital twins, comprising: The image processing module is used to preprocess images dynamically captured in an industrial environment using the image processing model on the embedded terminal, and input the preprocessed image with the edge features of the target object into the image database. The virtual model update program module is used to obtain the image with the edge features of the target object from the image database through the visual twin master kernel, and dynamically update the virtual model of the industrial environment in the model database. The training module is used by the visual twin master core to collect customized training data for the image processing model in the dynamically updated virtual model according to the customized requirements configuration information, and send the customized training data to the artificial intelligence training terminal to simultaneously collect the customized training data and train the image processing model. The dynamic software testing module is used to reload the image processing model trained by the artificial intelligence training terminal on the embedded terminal, so as to realize automated control training of machine vision in the industrial environment.

7. The system according to claim 6, wherein, The system also includes a model building program module, used for: The visual twin master core performs image post-processing on training images of an industrial environment input from an embedded terminal, wherein the image post-processing includes: cleaning and / or image segmentation and / or color gamut conversion and / or image compression and / or edge detection; The object types and features of each object in the post-processed training image are identified, the target object is determined, and the training image is compiled into a spatial image of a three-dimensional point cloud. The visual twin master kernel performs virtual mapping of the target object in the spatial image of the three-dimensional point cloud using machine vision, constructs a virtual model of the industrial environment, and stores the virtual model in the model database. The visual twin master core collects customized training data in the virtual model according to the customized requirements and configuration information, and sends the customized training data to the artificial intelligence training terminal for training the image processing model. The AI ​​training terminal loads the trained image processing model onto the embedded terminal.

8. The system according to claim 7, wherein, The model building program module is also used for: The spatial information of the industrial environment is determined by compiling the image data type of the target object in the spatial image of the three-dimensional point cloud using a visual twin kernel based on a real-time rendering engine. A virtual model of the industrial environment is constructed based on the spatial information.

9. An electronic device comprising: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the steps of the method according to any one of claims 1-5.

10. A storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 1-5.

Images