A simulation image generation method, a simulation image generation device and an electronic device

Abstract

A method for generating a simulated image, a device for generating a simulated image, and an electronic device. The present application discloses a method for generating a simulated image, a device, an electronic device, and a computer-readable storage medium. The method comprises: importing an environment to be tested on a preset simulation platform, wherein at least one camera and at least one environmental object are created in the environment to be tested, the environmental object comprises a target to be identified and an intelligent device to be tested, and the intelligent device to be tested comprises at least one camera; controlling the intelligent device to be tested and / or the target to be identified to move in the environment to be tested, and obtaining a processed image by controlling the parameters of the camera and / or performing image processing operations on the image output by the camera during this process; obtaining the attribute data of each environmental object in the processed image; and annotating the processed image based on the attribute data to obtain a simulated image. Through the scheme of the present application, a large number of high-fidelity simulated images that can be used to train visual algorithm models can be quickly obtained in a short time.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and in particular relates to a simulation image generation method, a simulation image generation device, an electronic device, and a computer-readable storage medium. Background Art

[0002] Visual algorithm models based on deep learning often require a large number of training data sets. Currently, large-scale publicly available data sets are usually used as training data sets to train visual algorithm models. However, for different needs and scenarios, large-scale publicly available data sets may not be able to cover them. This requires data collection based on specific needs, and then retraining or fine-tuning the visual algorithm model based on the collected data.

[0003] Currently, the method of data collection is usually to manually collect data in the field according to specific demand scenarios. However, field data collection often encounters complex collection conditions, which makes it difficult to collect the required data. In addition, the collected data may not have practical application value due to various factors. Therefore, how to quickly obtain data that can be used to train visual algorithm models has become a problem that needs to be solved urgently. Summary of the invention

[0004] The present application provides a simulation image generation method, a simulation image generation device, an electronic device and a computer-readable storage medium, which can quickly obtain a large number of high-fidelity simulation images that can be used to train visual algorithm models in a short period of time.

[0005] In a first aspect, the present application provides a method for generating a simulated image, comprising:

[0006] Importing a test environment into a preset simulation platform, wherein at least one camera and at least one environmental object are created in the test environment, the environmental object includes a target to be identified and an intelligent device to be tested, and the intelligent device to be tested includes at least one camera;

[0007] Controlling the intelligent device to be tested and / or the target to be identified to move in the environment to be tested;

[0008] When the above-mentioned intelligent device to be tested and / or the above-mentioned target to be identified moves in the above-mentioned environment to be tested, a processed image is obtained by controlling parameters of the above-mentioned camera and / or performing image processing operations on the image output by the above-mentioned camera;

[0009] Obtaining attribute data of each environmental object in the above processed image;

[0010] The processed image is annotated based on the attribute data to obtain a simulated image.

[0011] In a second aspect, the present application provides a simulation image generation device, comprising:

[0012] An import module, used for importing a test environment into a preset simulation platform, wherein at least one camera and at least one environmental object are created in the test environment, the environmental object includes a target to be identified and an intelligent device to be tested, and the intelligent device to be tested includes at least one camera;

[0013] A control module, used for controlling the above-mentioned intelligent device to be tested and / or the above-mentioned target to be identified to move in the above-mentioned environment to be tested;

[0014] A processing module, used for obtaining a processed image by controlling parameters of the camera and / or performing image processing operations on the image output by the camera when the smart device to be tested and / or the target to be identified moves in the environment to be tested;

[0015] An acquisition module, used to acquire attribute data of each environmental object in the above-mentioned processed image;

[0016] The annotation module is used to annotate the processed image based on the attribute data to obtain a simulated image.

[0017] In a third aspect, the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method of the first aspect when executing the computer program.

[0018] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method of the first aspect are implemented.

[0019] In a fifth aspect, the present application provides a computer program product, which includes a computer program. When the computer program is executed by one or more processors, it implements the steps of the method of the first aspect.

[0020] Compared with the prior art, the present application has the following beneficial effects: first, the environment to be tested is imported into a preset simulation platform, wherein at least one camera and at least one environmental object are created in the environment to be tested, the environmental object includes a target to be identified and an intelligent device to be tested, and the intelligent device to be tested includes at least one camera, and then the intelligent device to be tested and / or the target to be identified are controlled to move in the environment to be tested, and in this process, the processed image is obtained by controlling the parameters of the camera and / or performing image processing operations on the image output by the camera, and the attribute data of each environmental object in the processed image is obtained, and finally the processed image can be annotated based on the attribute data to obtain a simulated image. The above process does not require manual field data collection, but generates a simulated image by simulating the environment through a simulation platform, which can greatly save the time required for data collection. Moreover, when the demand changes, it only needs to change and import different environments to be tested, or perform different parameter control on the camera, or adopt different image processing operations, so as to obtain a simulated image that adapts to the change in demand, and has strong scalability and flexibility. It can be understood that the beneficial effects of the second to fifth aspects can refer to the relevant description in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0022] Figure 1 It is a schematic diagram of the implementation process of the simulation image generation method provided in the embodiment of the present application;

[0023] Figure 2 is an example diagram of a processed image obtained in an occlusion scene provided in an embodiment of the present application;

[0024] Figure 3 is an example diagram of a processed image obtained under a blurred perspective provided in an embodiment of the present application;

[0025] Figure 4 is an example diagram of a processed image obtained under a blurred close-up perspective provided in an embodiment of the present application;

[0026] Figure 5 is an example diagram of a processed image obtained in a dimly lit scene provided by an embodiment of the present application;

[0027] Figure 6 is an example diagram of a processed image obtained in a light overexposure scenario provided in an embodiment of the present application;

[0028] Figure 7 is a structural block diagram of a simulation image generating device provided in an embodiment of the present application;

[0029] Figure 8 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0031] In order to illustrate the technical solution proposed in this application, a specific embodiment is provided below for illustration.

[0032] The following describes a method for generating a simulated image provided by an embodiment of the present application. Figure 1 , the simulation image generation method comprises:

[0033] Step 101, importing the environment to be tested into a preset simulation platform.

[0034] In the embodiment of the present application, a simulation platform can be used to simulate the actual environment required for data collection. For the case where the intelligent device to be tested is a robot, as an example only, the simulation platform can be the Omniverse Isaac Sim simulation platform, which is a simulation platform for artificial intelligence (AI) robot development applications launched by NVIDIA. The Omniverse Isaac Sim simulation platform is integrated with real-time ray tracing technology (Ray Tracing, RTX), which can provide photo-level graphical rendering based on the visual sensor perspective; it is also integrated with a physics engine development kit PhysX SDK for rigid body kinematics calculation based on a graphics processing unit (graphics processing unit, GPU), which can provide fast and stable physical simulation. In addition, it can provide a data transmission interface for the robot operating system (Robot Operating System, ROS) and the robot software development kit Isaac SDK. In addition, users can also use Python-based scripting interfaces and extension systems to customize the interface and functions of the simulation platform.

[0035] For the Omniverse Isaac Sim simulation platform, the scene files used are specifically Pixar's Universal Scene Description (USD) and NVIDIA's Material Definition Library (MDL). Among them, USD is equipped with a rich scene representation interface (Application Programming Interface, API), which can directly give complex functional operations to the scene by modifying the API; MDL can be used to describe the physical material format of the real-time ray tracing physical rendering solution, which can ensure the basic consistency of materials when multiple applications interact. In addition, files in these two formats are easy to transmit, which effectively guarantees the service performance of remote file transmission.

[0036] When the electronic device imports the environment to be tested into the simulation platform, it should create at least one camera and at least one environmental object in the imported environment to be tested, wherein the environmental object includes a target to be identified and an intelligent device to be tested, and the intelligent device to be tested includes at least one camera. That is, there are two types of cameras involved in the environment to be tested, one is a camera created at certain fixed positions in the environment to be tested, and the other is a camera on the intelligent device to be tested. In fact, the intelligent device to be tested is also created by importing. For the convenience of explaining the scheme of the embodiment of the present application, the following will take the simulation platform as the Omniverse Isaac Sim simulation platform and the intelligent device to be tested as a robot as an example to explain each step.

[0037] The electronic device can import the USD file of the environment to be tested into the simulation platform to realize the import of the environment to be tested. As an example, the environment to be tested can be the environment of the RoboCup competition. Of course, according to the specific requirements of different projects, the USD file of the corresponding environment to be tested can be built and imported to realize the replacement of the environment to be tested.

[0038] The electronic device can also adjust the initial position data of the robot, write the initial position data as a parameter into the existing USD file of the robot (the USD file contains information such as the shape, material and appearance of each component of the robot, as well as information such as the speed and initial position of the robot), and import the USD file into the simulation platform to realize the import of the robot (that is, the intelligent device to be tested). In addition, the electronic device can also adjust the parameters of the robot's camera so that the parameters of the camera are consistent with the parameters of the ZED camera used by the real robot. Specifically, the parameters mainly include: focal length parameters, field of view size parameters, shutter parameters and other internal parameters, which are not limited here.

[0039] The use of USD files can be understood through the following explanation: the USD file of the environment to be tested is similar to the main function in the code. Each movable environmental object in the environment to be tested (such as a football and a robot, etc.) also has a corresponding USD file. The USD file of the environment to be tested can call other USD files, similar to calling other written functions in the main function. The fixed environmental objects in the environment to be tested (such as the goal and the lawn) are similar to the non-call declaration statements in the main function. From this, it can be understood that the initial position and other information of the object in the called USD file are similar to the parameters in the called function, which can be set to the default value or passed in for modification when the function is called.

[0040] Specifically, after importing the test environment, the test environment can be improved in the following ways:

[0041] One way is to set obstacles and occluders in the test environment displayed on the simulation platform, which can simulate the occlusion influence factor in visual target tracking. Figure 2 As shown, through the simulation platform, it can be simulated that the football (that is, the target to be identified) is blocked by the left person in the foreground.

[0042] Another way is to create multiple cameras at different positions and / or different angles in the test environment displayed on the simulation platform; of course, since the robot also has a camera, the camera on the robot can be called to simulate the recording of the dynamic motion of the target under different perspectives. Just as an example, when the test environment is the RoboCup competition, the robot's camera can be built in advance according to the requirements of the RoboCup competition. In the RoboCup environment displayed on the simulation platform, you only need to fine-tune the robot's camera and then call it directly. It can be understood that since the RoboCup competition limits the use of only the robot's left eye camera, no other cameras are added to the robot in this example. If other projects have specific needs, you can also add other cameras to the robot, thereby increasing the robot's different perspectives.

[0043] The third method is to build a wooden ramp with adjustable height in the test environment displayed on the simulation platform. When the test environment is the RoboCup competition, the target to be identified is a football, and then the football can be rolled down the ramp to obtain the initial velocity of the movement.

[0044] Step 102: Control the smart device to be tested and / or the target to be identified to move in the environment to be tested.

[0045] In an embodiment of the present application, in order to simulate various situations that may occur in an actual environment, the electronic device can control the robot and / or the target to be identified to move in the environment to be tested.

[0046] For the robot, the robot can be controlled to walk omnidirectionally and / or rotate its head motor in the test environment displayed on the simulation platform. On the one hand, the jitter of the robot during movement can simulate the blur jitter of the robot's camera during dynamic movement; on the other hand, the robot moving to different positions can simulate the influencing factors such as the target moving out of sight and scale change in visual target tracking.

[0047] For the target to be identified, the third method of improving the test environment described in step 101 can allow the target to be identified to obtain an initial velocity of motion, thereby making the target to be identified in a continuous motion state, which is conducive to the subsequent generation of richer simulation images.

[0048] Step 103 , when the smart device to be tested and / or the target to be identified moves in the environment to be tested, a processed image is obtained by controlling parameters of the camera and / or performing image processing operations on the image output by the camera.

[0049] When the intelligent device to be tested and / or the target to be identified moves in the environment to be tested, each camera can start working, that is, to collect images. In the simulation platform, various image processing operations can be performed to obtain processed images with different image effects. Of course, the parameters of the camera can also be adjusted to obtain processed images with different image effects. It can be understood that steps 102 and 103 are both for simulating images in non-ideal laboratory environments, such as: simulating images with problems such as blur or flickering caused by the shaking of the robot's camera due to the movement of the robot; simulating images with problems such as dim light or overexposure due to weather or direct light.

[0050] In one application scenario, the image processing operation on the image output by the camera may be: setting noise points in the image output by the camera, which may reduce the clarity of the processed image and increase the complexity.

[0051] In another application scenario, the image processing operation on the image output by the camera may be: adjusting the parameters contained in the image output by the camera that can affect the motion blur of the target to be identified. As an example only, the parameters may include a blur diameter parameter (Blur Diameter Fraction) and an exposure parameter (Exposure Fraction). In fact, the filter diameter can be adjusted by adjusting the values ​​of these two parameters, thereby adjusting the blur degree of the target to be identified. As an example only, the specific blur effect can be referred to Figure 3 and Figure 4 Football in. Figure 3 As shown, through the simulation platform, a football (i.e., the target to be identified) can be simulated. Figure 3The processed image is obtained under the blurred perspective when the image is moving rapidly (lower left of the figure). Figure 4 As shown, through the simulation platform, the processed image obtained under the blurred close-up perspective when the football (that is, the target to be identified) moves quickly can be simulated.

[0052] In another application scenario, the camera parameter control may be: controlling the camera shutter parameters. This can change the brightness of the image output by the camera, thereby obtaining processed images under different lighting conditions. Figure 5 As shown in the figure, the simulation platform can simulate the processed image obtained when the light is dark. Figure 6 As shown, the simulation platform can be used to simulate the processed image obtained when the light is overexposed. Of course, it is also possible to combine the control of the focal length parameter and / or field of view parameter of the camera to simulate the influence of factors such as scale change and motion blur on the image output by the camera to obtain the corresponding processed image.

[0053] Step 104, obtaining attribute data of each environmental object in the processed image.

[0054] In the embodiment of the present application, the Omniverse Isaac Sim simulation platform comes with a recording function SyntheticData Recorder, which can be recorded through a manual visualization window or directly run through a python script. Through this recording function, the processed image can be extracted, and the attribute data of each environmental object in the processed image can also be extracted. Specifically, the attribute data includes the state data and position data of the environmental object, wherein the state data is used to indicate whether the environmental object exists in the corresponding processed image, and the position data is used to indicate the coordinates of the environmental object in the corresponding processed image when the environmental object exists in the corresponding processed image.

[0055] As an example only, when the environment to be tested is the RoboCup competition, the environmental objects include not only football (i.e., the target to be identified), but also people, robots, goals and lawns, and even ceilings and ground blocked by lawns. Each environmental object has attribute data, and each environmental object can be located by a unique object number and different categories (each category has a unique category number). For example, the object number of the football is 0002, and the category is Ball; the object numbers of the goals on both sides are 0004 and 0012, respectively, and the category is Goal. The attribute data obtained by the electronic device is actually stored in a file in NYP format (i.e., .nyp file), and each line of the file corresponds to an environmental object in the environment to be tested. For a processed image, the format of the attribute data of a certain environmental object can be (object number, category number, x1, y1, x2, y2). Among them, x1 and y1 are the horizontal coordinates and vertical coordinates of the upper left vertex of the calibration rectangle of the environmental object; x2 and y2 are the horizontal coordinates and vertical coordinates of the lower right vertex of the calibration rectangle of the environmental object. If the environmental object is outside the field of view of the camera, x1, y1, x2 and y2 of the environmental object may be set to a predetermined fixed value to indicate that the environmental object does not exist in the corresponding processed image obtained based on the camera.

[0056] Just as an example, if the attribute data of a football extracted for a certain processed image is (1081, 2, 641, 189, 658, 207), then the meaning expressed by the attribute data is: the object number of the football in the test environment is 1081, the classification number is 2, and the corresponding category is Ball. In the processed image, the coordinates of the upper left vertex of the football's calibration rectangle are (641, 189), and the coordinates of the lower right vertex are (658, 207). The width and height of the football's calibration rectangle can be obtained through these two coordinates.

[0057] Step 105 , annotating the processed image based on the attribute data to obtain a simulated image.

[0058] In the embodiment of the present application, data cleaning and extraction operations can be performed on the attribute data, and the processed image can be annotated based on the attribute data after data cleaning and extraction, thereby obtaining accurate ground truth data required for training the visual algorithm model. It can be understood that the simulated image is a processed image with annotated data, that is, an annotated processed image. Since the annotation operation is performed by an electronic device, the time consumption of annotation can be reduced, and the occurrence of data annotation errors can be reduced.

[0059] Specifically, the processed image obtained in this embodiment is a file in PNG format (i.e., a .png file), and the obtained attribute data is a file in NPY format (i.e., a .npy file). In addition, the Omniverse Isaac Sim simulation platform can also provide depth information and object segmentation information, which will not be repeated here. It can be understood that each processed image (.png file) has corresponding attribute data of environmental objects (.npy file), that is, the .png file and the .npy file are one-to-one corresponding. In addition, the .png files and .npy files obtained based on the same camera are arranged and numbered in chronological order. The following is a brief description of the process of data cleaning and extraction operations:

[0060] Through the preset python script, the mixed .png files and .npy files obtained from the same camera can be separated, that is, the processed images and attribute data can be separated. The .png files can be stored in the preset image folder, and the .npy files can be stored in the preset data folder. It can be understood that all the processed images in the image folder are combined in time sequence, which is the video recorded by the Synthetic Data Recorder based on a camera; and the data folder stores the attribute data (that is, state data and position data) of all environmental objects in the video arranged in time sequence.

[0061] For each .npy file, based on the classification number of the environmental object to be extracted, the attribute data of the environmental object can be extracted. Of course, if the environmental object has multiple similar objects in the environment to be tested, the object number of the environmental object needs to be considered again. For example, if the environmental object to be extracted is a football, the classification number 2 of the football (or the object number 1081 of the football) can be used to extract the attribute data corresponding to the football from the .npy file. By subtracting the upper left vertex coordinates and the lower right vertex coordinates of the annotated rectangular box contained in the attribute data, the width w and height h of the target rectangular box can be obtained. Thus, the most common annotation format (x, y, w, h) in target tracking can be formed, representing the coordinates (x, y) of the upper left vertex of the target rectangular box and the side length (w, h) of the target rectangular box respectively. According to the order of the file number of .npy from small to large, the attribute data of the football in each .npy file in the data folder is extracted to form the annotation file (groundtruth file) of the football, and each line in the annotation file represents the position and size information of the football in each frame of the image processed in the time sequence.

[0062] It is understandable that in the case of manual data annotation, on the one hand, it is necessary for a person to perform operations such as visually identifying objects, determining the position of objects, and manually outlining the annotation rectangle of the object on the processed image one by one, and then extracting the position of the annotation rectangle in the annotated processed image to form an annotation file. In the case of a large amount of data, the manual annotation time increases accordingly, and long-term annotation may cause visual fatigue of the person, resulting in a high annotation error rate. On the other hand, since each operation in the annotation process depends on the person's own judgment, this may result in different people performing data annotation on the same processed image, and may obtain different target rectangles. And when encountering the situation of object occlusion, the probability of annotation error is even greater. The present application scheme generates simulation images based on the Omniverse Isaac Sim simulation platform, which basically eliminates the possibility of annotation errors, and takes very little time, greatly improving the efficiency of data collection.

[0063] In some embodiments, after step 105, the following operations may also be performed:

[0064] A1. Train the visual algorithm model to be trained through simulated images.

[0065] Since these simulated images have been labeled, the simulated images can be put into the training process of the visual algorithm model to be trained. Specifically, the simulated images can be classified first, for example, according to the classification ratio of 6:2:2, the simulated images can be randomly divided into training set, verification set and test set. The electronic device can thus train the visual algorithm model to be trained on a high-performance graphics processor through the training set, verification set and test set. It should be noted that the embodiments of the present application do not limit the specific type of the visual algorithm model; that is, steps A1 to A4 can be applied to any visual algorithm model.

[0066] A2. Perform an accuracy test on the trained visual algorithm model to obtain the accuracy of the trained visual algorithm model under at least one influencing factor.

[0067] After obtaining the trained visual algorithm model through step A1, the accuracy of the trained visual algorithm model can also be tested under at least one influencing factor to obtain the accuracy of the trained visual algorithm model under each influencing factor. It can be understood that the influencing factor can be set according to the specific application scenario of the visual algorithm model (ie, the training goal). For example, if the specific application scenario in the previous example is the RoboCup competition, the influencing factors can be set as follows: motion blur, illumination change, moving out of sight, camera movement, scale change, occlusion, rapid movement and rotation. Among them, rotation can be divided into in-plane rotation (such as tilting the head to the right or left) and out-of-plane rotation (such as lowering the head or raising the head).

[0068] In addition, the influencing factors may also include: deformation (for example, a person changes from standing to squatting), low resolution (generally refers to less than 800*600, the resolution used by the cameras in the RoboCup competition is generally 1280*720) and background clusters (for example, the background is a crossroads with people coming and going), etc. In this example, since the target to be identified is a football, deformation and rotation can be combined into one category, and there is no need to consider the deformation as an influencing factor separately. Since the application scenario is the RoboCup competition, the camera situation is basically stable, so the influencing factor of low resolution can be ignored. Since the competition area of ​​the competition is an artificial turf, and except for the robots of the two participating teams, the referee and the followers behind each robot, restrictions are imposed on other people in the field (no one else is allowed to enter), so the influencing factor of background clusters can be ignored.

[0069] A3. Detect whether the target influencing factor exists.

[0070] The electronic device may preset an accuracy threshold and compare the accuracy of the trained visual algorithm model under each influencing factor with the accuracy threshold. If the accuracy of the trained visual algorithm model under a certain influencing factor is lower than the accuracy threshold, the influencing factor may be determined as the target influencing factor.

[0071] A4. If there is a target influencing factor, the trained visual algorithm model is used as a new visual algorithm model to be trained, and based on the target influencing factor, the steps of controlling the movement of the intelligent device to be tested and / or the target to be identified in the test environment and subsequent steps are returned until there is no target influencing factor.

[0072] It can be understood that in the process of generating the simulated image, the image processing operation and the parameter control of the camera involved in step 103, as well as the movement of the intelligent device to be tested and the movement of the target to be identified involved in step 102 will all have an impact on the scene corresponding to the simulated image. It should be noted that the scene mentioned here does not refer to the environment represented by the image, but refers to the performance of the image under a certain influencing factor, such as the influencing factor of illumination change, which includes dim light scenes and overexposed light scenes, and the influencing factor of occlusion, which includes occlusion scenes, etc. Therefore, the operation of returning the steps and subsequent steps of controlling the movement of the intelligent device to be tested and / or the target to be identified in the test environment based on the target influence factor can be specifically expressed as follows: first, determine the target simulation image in the simulation image according to the target influence factor, that is, find the simulation image affected by the target influence factor as the target simulation image; then, since there may be multiple scenes under one target influence factor, it is necessary to analyze these target simulation images to determine in which scene under the target influence factor the accuracy of the visual algorithm model is lower, thereby determining the target scene based on the analysis results; thereafter, the steps and subsequent steps of controlling the movement of the intelligent device to be tested and / or the target to be identified in the test environment can be returned based on the target scene, that is, generate more new simulation data under the target scene through the simulation platform, and then train the new visual algorithm model to be trained according to the new simulation data until the trained visual algorithm model has high accuracy under various influencing factors.

[0073] As an example only, the lighting influencing factor may include the dim light scene, overexposed light scene and complex light change scene not shown in the previous text (such as flickering and changing light on the stage). Assume that through step A3, the target influencing factor is determined to be the lighting influencing factor; then, in step A4, it is further analyzed that the accuracy of the visual algorithm model in dim light scenes is low, and the accuracy in overexposed light scenes is relatively ideal; then the electronic device can generate a large number of new simulation images in dim light scenes through steps 101-105 to further train the visual algorithm model, and later the visual algorithm model can be made more accurate in both dim light scenes and overexposed light scenes, thereby improving the accuracy of the visual algorithm model under the lighting influencing factor.

[0074] As can be seen from the above, through the embodiments of the present application, there is no need for manual field data collection, but the environment is simulated through the simulation platform to generate a simulated image, which can greatly save the time required for data collection. Moreover, when the demand changes, it is only necessary to change and import different test environments, or to control different parameters of the camera, or to use different image processing operations to obtain simulated images that adapt to the changes in demand, which can make the simulated image generation method proposed in the embodiments of the present application have strong scalability and flexibility.

[0075] Corresponding to the simulation image generation method proposed above, the present application embodiment provides a simulation image generation device. Figure 7 , the simulation image generating device 700 in the embodiment of the present application includes:

[0076] The import module 701 is used to import the test environment into a preset simulation platform, wherein at least one camera and at least one environmental object are created in the test environment, the environmental object includes a target to be identified and an intelligent device to be tested, and the intelligent device to be tested includes at least one camera;

[0077] A control module 702 is used to control the intelligent device to be tested and / or the target to be identified to move in the environment to be tested;

[0078] The processing module 703 is used to obtain a processed image by controlling the parameters of the camera and / or performing image processing operations on the image output by the camera when the smart device to be tested and / or the target to be identified moves in the environment to be tested;

[0079] An acquisition module 704 is used to acquire attribute data of each environmental object in the above-mentioned processed image;

[0080] The annotation module 705 is used to annotate the processed image based on the attribute data to obtain a simulated image.

[0081] Optionally, the image processing operation on the image output by the camera includes:

[0082] Noise points are set in the image output by the above camera.

[0083] Optionally, the parameter control of the camera includes:

[0084] Control the shutter parameters of the above cameras.

[0085] Optionally, the image processing operation on the image output by the camera includes:

[0086] Adjust the parameters contained in the image output by the camera that can affect the motion blur of the target to be identified.

[0087] Optionally, the above-mentioned attribute data includes status data and position data of the environmental object, wherein the above-mentioned status data is used to indicate whether the environmental object exists in the corresponding processed image, and the above-mentioned position data is used to indicate the coordinates of the above-mentioned environmental object in the corresponding processed image when the environmental object exists in the corresponding processed image.

[0088] Optionally, the simulation image generating device 700 further includes:

[0089] A training module, used for training the visual algorithm model to be trained by using the simulated image after the simulated image is obtained;

[0090] A testing module, used to perform an accuracy test on the trained visual algorithm model to obtain the accuracy of the trained visual algorithm model under at least one influencing factor;

[0091] A detection module, used to detect whether a target influencing factor exists, wherein the accuracy of the trained visual algorithm model under the target influencing factor is lower than a preset accuracy threshold;

[0092] The replacement module is used to use the trained visual algorithm model as a new visual algorithm model to be trained if the above-mentioned target influencing factor exists, and trigger the operation of the control module 702 based on the above-mentioned target influencing factor until the above-mentioned target influencing factor does not exist.

[0093] Optionally, the scene corresponding to each processed image is determined based on the image processing operation corresponding to the processed image, the parameters of the camera corresponding to the processed image, the movement of the smart device to be tested and / or the movement of the target to be identified; the iterative module includes:

[0094] A first determining unit, configured to determine a target simulation image in the simulation image according to the target influencing factor;

[0095] A second determination unit is used to analyze the target simulation image and determine a target scene in the scene corresponding to the target simulation image according to the analysis result;

[0096] The alternation unit is used to trigger the operation of the control module 702 based on the above target scene.

[0097] As can be seen from the above, through the embodiments of the present application, there is no need for manual field data collection, but the environment is simulated through the simulation platform to generate a simulated image, which can greatly save the time required for data collection. Moreover, when the demand changes, it is only necessary to change and import different test environments, or to control different parameters of the camera, or to use different image processing operations to obtain simulated images that adapt to the changes in demand, which can make the simulated image generation method proposed in the embodiments of the present application have strong scalability and flexibility.

[0098] The present application also provides an electronic device. Figure 8 The electronic device 8 in the embodiment of the present application includes: a memory 801, one or more processors 802 ( Figure 8 Only one is shown) and a computer program stored in the memory 801 and executable on the processor. The memory 801 is used to store software programs and units, and the processor 802 executes various functional applications and data processing by running the software programs and units stored in the memory 801 to obtain the resources corresponding to the above-mentioned preset events. Specifically, the processor 802 implements the following steps when running the above-mentioned computer program stored in the memory 801:

[0099] Importing a test environment into a preset simulation platform, wherein at least one camera and at least one environmental object are created in the test environment, the environmental object includes a target to be identified and an intelligent device to be tested, and the intelligent device to be tested includes at least one camera;

[0100] Controlling the intelligent device to be tested and / or the target to be identified to move in the environment to be tested;

[0101] When the above-mentioned intelligent device to be tested and / or the above-mentioned target to be identified moves in the above-mentioned environment to be tested, a processed image is obtained by controlling parameters of the above-mentioned camera and / or performing image processing operations on the image output by the above-mentioned camera;

[0102] Obtaining attribute data of each environmental object in the above processed image;

[0103] The processed image is annotated based on the attribute data to obtain a simulated image.

[0104] Assuming that the above is the first possible implementation, in a second possible implementation provided on the basis of the first possible implementation, the image processing operation on the image output by the camera includes:

[0105] Noise points are set in the image output by the above camera.

[0106] In a third possible implementation manner provided on the basis of the first possible implementation manner, the parameter control of the camera includes:

[0107] Control the shutter parameters of the above cameras.

[0108] In a fourth possible implementation manner provided on the basis of the first possible implementation manner, the image processing operation on the image output by the camera includes:

[0109] Adjust the parameters contained in the image output by the camera that can affect the motion blur of the target to be identified.

[0110] In a fifth possible implementation provided as a basis on the first possible implementation, the attribute data includes status data and position data of environmental objects, wherein the status data is used to indicate whether the environmental object exists in the corresponding processed image, and the position data is used to indicate the coordinates of the environmental object in the corresponding processed image when the environmental object exists in the corresponding processed image.

[0111] In a sixth possible implementation provided on the basis of the first possible implementation, or the second possible implementation, or the third possible implementation, or the fourth possible implementation, or the fifth possible implementation, after obtaining the simulation image, the processor 802 further implements the following steps by running the computer program stored in the memory 801:

[0112] The visual algorithm model to be trained is trained by using the above-mentioned simulation images;

[0113] Performing an accuracy test on the trained visual algorithm model to obtain the accuracy of the trained visual algorithm model under at least one influencing factor;

[0114] Detecting whether a target influencing factor exists, wherein the accuracy of the trained visual algorithm model under the target influencing factor is lower than a preset accuracy threshold;

[0115] If the above-mentioned target influencing factor exists, the above-mentioned trained visual algorithm model will be used as a new visual algorithm model to be trained, and based on the above-mentioned target influencing factor, the above-mentioned steps of controlling the above-mentioned intelligent device to be tested and / or the above-mentioned target to be identified to move in the above-mentioned test environment and subsequent steps will be returned until the above-mentioned target influencing factor no longer exists.

[0116] In a seventh possible implementation provided on the basis of the sixth possible implementation, the scene corresponding to each processed image is determined based on the image processing operation corresponding to the processed image, the parameters of the camera corresponding to the processed image, the movement of the smart device to be tested and / or the movement of the target to be identified; the step of returning to execute the step of controlling the movement of the smart device to be tested and / or the target to be identified in the test environment based on the target influence factor and subsequent steps include:

[0117] Determining a target simulation image in the simulation image according to the target influencing factor;

[0118] Analyze the target simulation image, and determine the target scene in the scene corresponding to the target simulation image according to the analysis result;

[0119] Based on the target scene, the step of controlling the intelligent device to be tested and / or the target to be identified to move in the environment to be tested and subsequent steps are executed.

[0120] It should be understood that in the embodiment of the present application, the processor 802 may be a CPU, and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0121] The memory 801 may include a read-only memory and a random access memory, and provide instructions and data to the processor 802. A part or all of the memory 801 may also include a non-volatile random access memory. For example, the memory 801 may also store information of a device category.

[0122] As can be seen from the above, through the embodiments of the present application, there is no need for manual field data collection, but the environment is simulated through the simulation platform to generate a simulated image, which can greatly save the time required for data collection. Moreover, when the demand changes, it is only necessary to change and import different test environments, or to control different parameters of the camera, or to use different image processing operations to obtain simulated images that adapt to the changes in demand, which can make the simulated image generation method proposed in the embodiments of the present application have strong scalability and flexibility.

[0123] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the above-mentioned device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0124] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0125] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of external device software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0126] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the system embodiments described above are merely schematic. For example, the division of the above modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0127] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0128] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the associated hardware through a computer program. The above-mentioned computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the above-mentioned computer program includes computer program code, and the above-mentioned computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The above-mentioned computer-readable storage medium may include: any entity or device capable of carrying the above-mentioned computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer-readable memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the above-mentioned computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable storage media does not include electrical carrier signals and telecommunication signals.

[0129] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for generating a simulation image, characterized in that: include: Importing a test environment into a preset simulation platform, wherein at least one camera and at least one environmental object are created in the test environment, the environmental object includes a target to be identified and an intelligent device to be tested, and the intelligent device to be tested includes at least one camera; Controlling the intelligent device to be tested and / or the target to be identified to move in the environment to be tested; When the intelligent device to be tested and / or the target to be identified moves in the environment to be tested, obtaining a processed image by controlling parameters of the camera and / or performing image processing operations on the image output by the camera; Acquire attribute data of each environmental object in the processed image; Annotating the processed image based on the attribute data to obtain a simulated image; Training the visual algorithm model to be trained by using the simulated image; Performing an accuracy test on the trained visual algorithm model to obtain the accuracy of the trained visual algorithm model under at least one influencing factor; Detecting whether a target influencing factor exists, wherein the accuracy of the trained visual algorithm model under the target influencing factor is lower than a preset accuracy threshold; If the target influence factor exists, the trained visual algorithm model is used as a new visual algorithm model to be trained, and based on the target influence factor, the steps of controlling the movement of the intelligent device to be tested and / or the target to be identified in the test environment and subsequent steps are returned until the target influence factor no longer exists.

2. The method for generating a simulation image according to claim 1, wherein: The image processing operation on the image output by the camera includes: Noise points are set in the image output by the camera.

3. The method for generating a simulation image according to claim 1, wherein: The parameter control of the camera includes: The shutter parameters of the camera are controlled.

4. The method for generating a simulation image according to claim 1, wherein: The image processing operation on the image output by the camera includes: The parameters contained in the image output by the camera that can affect the motion blur of the target to be identified are adjusted.

5. The method for generating a simulation image according to claim 1, wherein: The attribute data includes state data and position data of the environmental object, wherein the state data is used to indicate whether the environmental object exists in the corresponding processed image, and the position data is used to indicate the coordinates of the environmental object in the corresponding processed image when the environmental object exists in the corresponding processed image.

6. The method for generating a simulation image according to claim 1, wherein: The scene corresponding to each processed image is determined based on the image processing operation corresponding to the processed image, the parameters of the camera corresponding to the processed image, the movement of the smart device to be tested and / or the movement of the target to be identified; the step of returning to execute the step of controlling the movement of the smart device to be tested and / or the target to be identified in the test environment based on the target influence factor and subsequent steps include: Determining a target simulation image in the simulation image according to the target influencing factor; Analyzing the target simulation image, and determining a target scene in the scene corresponding to the target simulation image according to the analysis result; Based on the target scene, the step of controlling the intelligent device to be tested and / or the target to be identified to move in the environment to be tested and subsequent steps are executed.

7. A simulation image generating device, characterized in that: include: An import module, used for importing a test environment into a preset simulation platform, wherein at least one camera and at least one environmental object are created in the test environment, the environmental object includes a target to be identified and an intelligent device to be tested, and the intelligent device to be tested includes at least one camera; A control module, used for controlling the intelligent device to be tested and / or the target to be identified to move in the environment to be tested; A processing module, used for obtaining a processed image by controlling parameters of the camera and / or performing image processing operations on the image output by the camera when the intelligent device to be tested and / or the target to be identified moves in the environment to be tested; An acquisition module, used for acquiring attribute data of each environmental object in the processed image; A labeling module, used for labeling the processed image based on the attribute data to obtain a simulated image; A training module, used for training the visual algorithm model to be trained by using the simulated image after the simulated image is obtained; A testing module, used to perform an accuracy test on the trained visual algorithm model to obtain the accuracy of the trained visual algorithm model under at least one influencing factor; A detection module, used to detect whether a target influencing factor exists, wherein the accuracy of the trained visual algorithm model under the target influencing factor is lower than a preset accuracy threshold; The iteration module is used to use the trained visual algorithm model as a new visual algorithm model to be trained if the target influencing factor exists, and trigger the operation of the control module based on the target influencing factor until the target influencing factor does not exist.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images