A data annotation method, device and storage medium based on three-dimensional simulation scene
By using time segmentation processing and shader technology in three-dimensional simulation scenarios, pixel-level annotation of image frames is solved, and the problem of low image labeling efficiency in three-dimensional simulation scenarios in the prior art is achieved, and the accurate reflection and structure of lighting changes is achieved. Generate labeled data.
Patent Information
- Application Number
- CN202110442147.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-04-23
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2041-04-23
AI Technical Summary
In the prior art, the pixel-level labeling efficiency of images in three-dimensional simulation scenarios is low and cannot effectively adapt to the training needs of various algorithms. It is difficult to achieve accurate data labeling when the light changes.
By obtaining a three-dimensional simulation scene, the image frame is colored according to different shading strategies, the area within the lighting range is processed in time segments, pixel-level overlay annotation is performed, the lighting coloration results are generated, and the vertex and cel colorers are used to measure the target attribute parameters. Perform staining to generate structured labeled data.
Image annotation adapted to various algorithm training is realized, ensuring accurate reflection of lighting changes, improving labeling efficiency and accuracy, and the generated labeling data can reflect lighting time.
Smart Images

Figure CN113223146B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing technology, and in particular to a data annotation method, device and storage medium based on a three-dimensional simulation scene. Background Art
[0002] With the development of virtual reality and three-dimensional rendering technology, virtual scenes can already be so realistic that they are indistinguishable from the real thing. In addition to providing an immersive experience, three-dimensional simulation scenes can also serve as a source of training data for related algorithms such as computer vision. The advantages of using three-dimensional simulation scenes instead of actual scene shooting are: it can greatly reduce the workload of manual data collection; it can achieve automatic data annotation that cannot be achieved in real-life scenes; and it can be combined with simulation algorithms to generate data under special events. Existing training data sets are usually obtained by manually acquiring the images to be detected and annotating them manually, which is labor-intensive and inefficient. Methods for generating annotated data using three-dimensional simulation scenes have yet to be developed. Summary of the invention
[0003] The problem solved by the present invention is how to annotate images obtained in a three-dimensional simulation scene at multiple angles at the pixel level.
[0004] To solve the above problems, the present invention provides a data annotation method based on a three-dimensional simulation scene, comprising:
[0005] Acquire a three-dimensional simulation scene, extract image frames, and preprocess the image frames to obtain the original scene coloring result; acquire the annotation type and its corresponding coloring strategy, and acquire the target attribute parameters of each pixel of the target in the image frame according to the annotation type; color the image frame according to the target attribute parameters and the coloring strategy to obtain the coloring result; wherein the annotation type includes illumination attributes, and the coloring strategy corresponding to the illumination attributes includes: dividing the time into segments, taking time periods as units, uniformly processing the image frames of each time period, annotating the areas within the illumination range, and superimposing the annotation data at the corresponding pixels of the image frames belonging to the same time period to obtain the illumination coloring result.
[0006] Compared with the prior art, the present invention obtains a three-dimensional simulation scene, obtains the corresponding annotation type based on different coloring strategies, and then colors the image frames according to the coloring strategies, thereby ensuring that the annotated image frames can adapt to the training of various algorithms; by superimposing and annotating the area of the illumination range in pixels to obtain the illumination coloring result, it can ensure that a thermal map reflecting the illumination time is accurately obtained.
[0007] Optionally, the annotation includes annotating the position and range of the lighting; the time is segmented, the image frames in each time period are uniformly processed in units of time periods, the area within the lighting range is annotated, and the annotation data at the corresponding pixels of the image frames belonging to the same time period are superimposed to obtain the lighting shading result, including: segmenting the time period with lighting in the three-dimensional simulation scene, calculating the lighting range of each image frame belonging to the same time period, and rendering the area within the lighting range into a first color; counting the number of times each pixel of each image frame belonging to the same time period is rendered into the first color; calculating a heat map based on the number of times, superimposing the heat map back to the first image frame of each time period, and rendering to obtain the lighting shading result.
[0008] Therefore, by segmenting the lighting time period, more detailed lighting shading results can be obtained, and by coloring the lighting area, a heat map is obtained, and then the lighting shading results are obtained. Since the image frames of the lighting map are taken from a static three-dimensional simulation scene, the heat map is superimposed on the first image frame of the time period to ensure accurate lighting shading results.
[0009] Optionally, according to the target attribute parameters and in accordance with the shading strategy, the image frame is shading to obtain a shading result, including: shading the image frame using the vertex shader based on the shading strategy to obtain a vertex shading image; using the fragment shader to obtain the shading result based on the vertex shading image, the shading strategy and the target attribute parameters, wherein the target attribute parameters include at least one of the following: type parameter, speed parameter, acceleration parameter, depth parameter, height parameter, color parameter of the target.
[0010] Therefore, the pixels in the image frame are dyed respectively according to the type parameter, speed parameter, acceleration parameter, depth parameter, height parameter and color parameter of the target in the image frame to generate dyeing results corresponding to the respective attributes, which are used to generate the corresponding annotated image.
[0011] Optionally, the performing shading using the vertex shader based on the shading strategy to color the image frame to obtain a vertex shaded image includes:
[0012] The vertex shader is used to shade the target according to the scene information of the image frame to obtain the vertex shaded image, wherein the scene information includes the viewpoint position and the texture normal.
[0013] Therefore, the vertex shader shades the vertices based on the viewpoint position and the texture normal to ensure that the shading result of the vertex is obtained.
[0014] Optionally, the using the fragment shader to obtain the shading result based on the vertex shading image, the shading strategy and the target attribute parameter comprises:
[0015] The fragment shader is used to obtain a texture color, and the texture color is superimposed on the vertex shaded image, wherein the texture color is obtained by sampling from texture pixels based on texture coordinates.
[0016] Therefore, the fragment shader processes the vertex shading result, obtains the texture color, and performs subsequent processing on the image frame through the texture color.
[0017] Optionally, when the annotation type is a panoramic segmentation attribute, the method of using the fragment shader to obtain the texture color, and superimposing the texture color on the vertex shading image, further includes: assigning a category and an instance ID to each pixel in the image frame based on the object tag in the image frame, assigning a different color to each object, and generating a panoramic segmentation shading result; when the annotation type is a depth attribute, the method of using the fragment shader to obtain the texture color, and superimposing the texture color on the vertex shading image, further includes: calculating the distance between the pixel point in the image frame and the viewpoint, normalizing the distance and generating a depth annotation shading result; when the annotation type is a speed attribute, the fragment shader is used to obtain the texture color, and after the texture color is superimposed on the vertex shading image, it also includes: based on the speed of the target in the image frame, the object of the image frame is marked, and the speed scalar is used as the standard for coloring to generate a speed annotation shading result; when the annotation type is a height attribute, the fragment shader is used to obtain the texture color, and after the texture color is superimposed on the vertex shading image, it also includes: based on the height of the pixel point in the image frame corresponding to the three-dimensional simulation scene, coloring to generate a height annotation shading result.
[0018] Therefore, when the annotation types are different, the corresponding attribute information and coloring strategies are obtained respectively, and the image frame is rendered and colored to obtain the corresponding coloring results.
[0019] Optionally, the output image includes: structured annotation data, including the category, instance ID, distance from the viewpoint, height from the ground, illumination time, and speed, acceleration, position, and direction angle of each pixel in the image frame.
[0020] In this way, it is guaranteed that images with structured annotated data are obtained.
[0021] Optionally, before calculating the illumination range of each image frame belonging to the same time period, the method further comprises:
[0022] The original scene rendering result and the first image frame of each time period are stored in a temporary queue.
[0023] Therefore, the original scene shading result and the first image frame of each time period are stored in a temporary queue to be called during subsequent processing, thereby ensuring that the efficiency of subsequent processing is improved.
[0024] The present invention also proposes a data annotation device based on a three-dimensional simulation scene, comprising:
[0025] A preprocessing module is used to obtain a three-dimensional simulation scene, extract image frames, and preprocess the image frames to obtain the original scene coloring result; an input module is used to obtain a labeling type and its corresponding coloring strategy, and obtain the target attribute parameters of each pixel of the target in the image frame according to the labeling type; a coloring module is used to color the image frame according to the target attribute parameters and the coloring strategy to obtain a coloring result; wherein the labeling type includes an illumination attribute, and the coloring strategy corresponding to the illumination attribute includes: dividing the time into segments, taking time periods as units, processing the image frames of each time period, labeling the areas within the illumination range, and superimposing the labeling data at the corresponding pixels of the image frames belonging to the same time period to obtain an illumination coloring result; an output module is used to select the corresponding coloring result according to demand and superimpose it with the original scene coloring result to obtain an output image.
[0026] The advantages of the data annotation device based on the three-dimensional simulation scene over the prior art are the same as those of the three-dimensional simulation scene data annotation method, which will not be described in detail here.
[0027] The present invention also proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is read and executed by a processor, the data labeling method based on the three-dimensional simulation scene as described above is implemented.
[0028] The advantages of the computer-readable storage medium over the prior art and the data annotation method of the three-dimensional simulation scene are the same, which will not be described in detail here. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] Figure 1 A schematic diagram of an embodiment of a data annotation method for a three-dimensional simulation scene according to the present invention;
[0030] Figure 2 It is a schematic diagram of an embodiment of the data annotation method of a three-dimensional simulation scene after the step S300 of the present invention is refined;
[0031] Figure 3 A flow chart of an embodiment of a data annotation method for a three-dimensional simulation scene according to the present invention;
[0032] Figure 4 A schematic diagram of an embodiment of a data annotation method for a three-dimensional simulation scene according to the present invention;
[0033] Figure 5 A schematic diagram of another embodiment of the data annotation method for a three-dimensional simulation scene of the present invention;
[0034] Figure 6 It is a schematic diagram of another embodiment of the data annotation method of the three-dimensional simulation scene of the present invention. DETAILED DESCRIPTION
[0035] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.
[0036] The present invention proposes a data annotation method based on a three-dimensional simulation scene. Figure 1 As shown, including:
[0037] Step S100, acquiring a three-dimensional simulation scene, extracting image frames, and preprocessing the image frames to obtain an original scene coloring result.
[0038] The image frame is preprocessed to obtain the original scene shading result, that is, the basic scene picture is obtained through preprocessing for subsequent rendering.
[0039] In one embodiment, the preprocessing is performed by preliminarily marking the image frame to serve as the image basis for the subsequent output image.
[0040] In one embodiment, the three-dimensional simulation scene performs scene modeling and scenario setting for the traffic monitoring scene, and then in the process of urban traffic operation simulation, viewpoints are selected to generate and annotate image data.
[0041] The three-dimensional simulation scene can be a simulation scene with continuous frames, that is, a dynamic three-dimensional scene. Therefore, in one embodiment, continuous image frames are extracted from the three-dimensional scene for preprocessing and saved to a temporary queue for marking dynamic targets, with the focus on marking the changes of targets in continuous image frames or the state of targets in image frames.
[0042] In another embodiment, image frames are extracted based on a three-dimensional simulation scene. The image frames are discrete image frames and are used to annotate image segmentation data, with a focus on annotating information of targets in fixed images.
[0043] In one embodiment, the image frame is a complete view of a certain angle in the three-dimensional scene image, that is, a complete image observed from this angle. By using the complete view as the image frame, more information can be obtained in one image frame. In another embodiment, only annotation information of a specific target is required, so the image frame is an image sub-block captured from the complete view, ensuring that more accurate annotations are provided when used for subsequent deep learning training, thereby improving the computational efficiency of deep learning and reducing the amount of computation.
[0044] Optionally, before the image frame is preprocessed, the following steps are included: defining target types, including roads, sidewalks, zebra crossings, cars, buses, trucks, pedestrians, guardrails, signs, and buildings.
[0045] The purpose of defining a target is to specify the scope and type of the target to be processed. In one embodiment, if an object does not belong to the defined target, the object is classified as background and no data is marked for it.
[0046] Preprocessing, including pre-rendering, is to pre-render the current image frame and save a copy to a temporary queue for subsequent processing. When this image frame needs to be called, the image is retrieved from the temporary queue for further processing.
[0047] Optionally, the preprocessing the image frame includes:
[0048] Acquire the image frame, and extract pixel information of the target in the image frame;
[0049] The image frame is preprocessed to obtain the original scene coloring result.
[0050] Optionally, before extracting the pixel information of the target in the image frame, the method includes: determining whether the pixel belongs to a preset target.
[0051] Preprocessing the image frame also includes extracting pixel information of the target in the image frame, that is, extracting information of the target where the pixel is located in units of pixels. For example, in one embodiment, Figure 3 As shown, the process of extracting pixel information is to determine whether the current pixel is a specified target. If so, the information of the current pixel is extracted, including the distance between the pixel and the viewpoint, the height of the z-axis, the object category, the instance number of the object, the speed and the center of mass position of the object, the type of the object, the speed information, the acceleration information, the depth information, and the height information. The distance between the pixel and the viewpoint and the center of mass position are calculated in the three-dimensional simulation scene. The pixel information is used to provide information data for subsequent shader processing.
[0052] The image frame is preprocessed to obtain the original scene shading result, which is used for subsequent rendering and superimposed with the subsequent shading result to form the final output image. At the same time, the properties of all objects required for subsequent rendering are also extracted and transmitted as the data basis for subsequent processing.
[0053] In one embodiment, the image frame is not further rendered, and only the original rendering result is output as the output image.
[0054] Step S200: obtaining a marking type and a corresponding coloring strategy, and obtaining a target attribute parameter of each pixel of the target in the image frame according to the marking type.
[0055] The matrix, vertices, normals, and attribute information in the image frame are sent to the GPU, and the matrix, vertices, and normals are used to calculate the vertex shaded image; the vertex shader and the fragment shader are rendered and shaded in the GPU.
[0056] Optionally, a vertex shader is used to determine the color of a point by calculating the vertex's color based on the viewpoint's position in the image frame, the texture normal, and the lighting.
[0057] Optionally, a fragment shader obtains a texture color and superimposes it on the vertex shading image, and obtains a corresponding shading result according to different attributes, wherein the texture color is obtained by sampling from texture pixels based on texture coordinates.
[0058] The fragment shader fills the vertex shading image with color. Since the preprocessing in step S100 provides a variety of information, different strategies are used to color different information separately to generate shading results corresponding to the information type. For example, in one embodiment, when rendering the height above the ground shading result and the speed shading result, the target height information transmitted by the preprocessing is used to render the target as different colors based on different heights; the target speed information transmitted by the preprocessing is used to render the target as different colors based on different speeds.
[0059] The annotation type is selected from the target attribute parameters of the pixel. The attribute in the corresponding image frame is selected as the attribute type to be annotated according to the needs, and then the corresponding coloring strategy is obtained. Each attribute has its own coloring strategy. The target attribute parameters of each pixel of the target are obtained according to the attribute type to be annotated, and the target attribute parameters of each pixel of the non-target can also be obtained.
[0060] In one embodiment, the attribute type to be annotated is a depth map, and the depth in the image frame is selected as the attribute type to be annotated according to the requirements, and then the depth shading strategy is obtained, and the corresponding color is rendered for all pixels in the image frame according to the depth. Therefore, in this embodiment, the depth data of all pixels in the image frame is obtained, different colors are determined according to different depths, and finally the image frame is rendered.
[0061] Step S300: coloring the image frame according to the target attribute parameters and the coloring strategy to obtain a coloring result.
[0062] Among them, Figure 4 As shown (the attached figure is an image after grayscale processing), the annotation type includes illumination attributes, and the coloring strategy corresponding to the illumination attribute includes: segmenting the time, taking the time period as the unit, uniformly processing the image frames of each time period, annotating the area within the illumination range, and superimposing the annotation data at the corresponding pixels of the image frames belonging to the same time period to obtain the illumination coloring result;
[0063] Annotation is to color the image frame. The image is colored using the RGB color space to obtain a labeled image. The color data on the image is superimposed to obtain an image with superimposed data, that is, a light coloring result, which can reflect the length of time of illumination in each area of the image frame.
[0064] The shading result includes a lighting shading result, and the lighting shading result is obtained by processing multiple image frames in the same viewing angle.
[0065] Select the corresponding shading result according to the needs and superimpose it with the original scene shading result to obtain the output image.
[0066] The multiple aspects of information contained in the same image frame are colored respectively to obtain coloring results corresponding to the information.
[0067] Optionally, the annotation includes annotating the position and range of the lighting; segmenting the time, uniformly processing the image frames within each time period, annotating the area within the lighting range, and superimposing the annotation data at the corresponding pixels of the image frames belonging to the same time period to obtain the lighting shading result includes: segmenting the time period with lighting in the three-dimensional simulation scene, calculating the lighting range of each image frame belonging to the same time period, and rendering the area within the lighting range into a first color; counting the number of times each pixel of each image frame belonging to the same time period is rendered into the first color; calculating a heat map based on the number of times, superimposing the heat map back to the first image frame of each time period, and rendering to obtain the lighting shading result.
[0068] Before calculating the illumination range of each image frame belonging to the same time period, it includes: storing the original scene shading result and the first image frame of each time period in a temporary queue.
[0069] The rendering process of the illumination shading result is different from that of other shading results. The main difference is that other rendering results use one image frame to directly color the image frame, while the illumination shading result superimposes the illumination annotation data of the image frames within a continuous time period to form an illumination map. The illumination map can intuitively display the illumination time length of each area in the image frame.
[0070] In one embodiment, the time of the day is segmented, and every two hours is taken as a time period. All image frames in each time period are obtained, and the first image frames of all time periods are stored in a temporary queue as the image carrier of the coloring result. The illumination range of all image frames from zero to two points is calculated based on the light source, and the area within the illumination range is rendered white, and the non-illuminated area is rendered black. Then, the heat map is calculated based on the number of times black or white is rendered, and superimposed on the first image frame of zero point to form a light map. Similarly, the light maps of other time periods are obtained.
[0071] Optionally, the light maps are superimposed to form a light shading result.
[0072] Taking the above embodiment as an example, in another embodiment, the illumination maps of all time periods of a day are superimposed as illumination shading results, and the illumination shading results are superimposed on the first image frame at zero point.
[0073] Alternatively, if Figure 5 As shown (the attached figure is an image after grayscale processing), according to the target attribute parameters and the coloring strategy, the image frame is colored, and the following coloring results are also included:
[0074] When the annotation type is a panoramic segmentation attribute, the step of obtaining a texture color using the fragment shader and superimposing the texture color onto the vertex-shaded image further includes: assigning a category and an instance ID to each pixel in the image frame based on an object tag in the image frame, assigning a different color to each object, and generating a panoramic segmentation shading result;
[0075] When the annotation type is a depth attribute, the step of using the fragment shader to obtain a texture color and superimposing the texture color onto the vertex shaded image further includes: calculating a distance between a pixel point in the image frame and a viewpoint, normalizing the distance and generating a depth annotation shaded result;
[0076] When the annotation type is a speed attribute, the step of using the fragment shader to obtain a texture color and superimposing the texture color onto the vertex shaded image further includes: marking the object of the image frame based on the speed of the target in the image frame, coloring the object based on the speed scalar, and generating a speed annotation coloring result;
[0077] When the annotation type is a height attribute, the method further includes: coloring the height in the three-dimensional simulation scene based on the pixel points in the image frame corresponding to the height attribute, and generating a height annotation coloring result.
[0078] In one embodiment, when the annotation type is a panoramic segmentation attribute, a depth attribute, a speed attribute, or a height attribute, an image frame is obtained, the image frame is preprocessed and stored in a temporary queue, the image frame is obtained from the temporary queue based on the annotation type and its corresponding coloring strategy, and the image frame is colored according to the corresponding coloring strategy. In this embodiment, all attributes are colored separately to obtain panoramic segmentation coloring results, depth coloring results, speed coloring results, and height coloring results, and the above four coloring results are all colored based on a single image frame.
[0079] like Figure 5 As shown, from left to right are an example of a depth annotation map, a velocity annotation map, and a height annotation map.
[0080] The corresponding target attribute parameters are obtained respectively from step S200 to color the image frames respectively.
[0081] In one embodiment, after extracting the image frames and preprocessing the image frames, the image frames are colored to obtain illumination coloring results, panoramic segmentation coloring results, depth annotation coloring results, speed annotation coloring results, and height annotation coloring results, respectively, and are stored in a temporary queue, waiting to be superimposed with the original coloring results and then output.
[0082] Optionally, the coloring result is not an outputtable coloring result, and is superimposed with the original coloring result to generate the final image pixel color as the output result.
[0083] Alternatively, if Figure 2 As shown, step S300 includes:
[0084] Step S301 : coloring the image frame using the vertex shader based on the coloring strategy to obtain a vertex colored image.
[0085] Vertex shaders are used to describe operations performed on vertices, such as coordinate transformation, calculating lighting formulas to generate the color of each vertex, and calculating texture coordinates. Vertex shaders can increase the speed of rendering scenes, perform vertex shading calculations based on the viewpoint position, map normals, and lighting of the image frame, and color the vertices within the image frame to obtain a vertex-shaded image.
[0086] In one embodiment, the vertex shader is used to perform vertex shading on the image frame, obtain a vertex shading image, and send the vertex shading image to the fragment shader; calculate texture coordinates, and provide the texture coordinates to the fragment shader for further rendering and shading.
[0087] According to the current rendering mode, different attribute information is obtained from the GPU for corresponding coloring. In one embodiment, the current rendering mode is the depth annotation mode, so the depth value of the pixel of the target image frame is obtained from the GPU, and the depth value includes the distance of the pixel from the viewpoint. Then, the target in the image frame is dyed using a preset dyeing strategy to obtain a depth annotation coloring result.
[0088] Optionally, the vertex shader is used to shade an object according to scene information of the image frame to obtain the vertex shaded image, wherein the scene information includes a viewpoint position and a texture normal.
[0089] Optionally, the fragment shader is used to obtain a texture color, and the texture color is superimposed on the vertex shading image, wherein the texture color is obtained by sampling from texture pixels based on texture coordinates.
[0090] The fragment shader combines texture coordinates and colors to further fill and color the vertex shading image. It first obtains the texture color in the image frame, and then fills the image frame with color according to the current rendering mode to obtain a shaded image.
[0091] Step S302, using the fragment shader, obtains the shading result based on the vertex shading image, the shading strategy and the target attribute parameters, wherein the target attribute parameters include at least one of the following: type parameter, speed parameter, acceleration parameter, depth parameter, height parameter, and color parameter of the target.
[0092] Step S400: Select a corresponding coloring result according to the requirement and superimpose it with the original scene coloring result to obtain an output image.
[0093] When it is necessary to obtain a colored image, the processed coloring result is superimposed with the original scene coloring to obtain the output image output.
[0094] In one embodiment, if Figure 3As shown, if a depth image needs to be output, the processed depth shading result is superimposed with the original scene shading to generate the final image pixel color.
[0095] Optionally, the output image includes: structured annotation data, including the category, instance ID, distance from the viewpoint, height from the ground, illumination time, and speed, acceleration, position, and direction angle of each pixel in the image frame.
[0096] The content of structured annotation data includes all annotation categories of the 3D simulation scene data annotation method, and can comprehensively obtain all data types within the image frame to support the training of machine learning algorithms.
[0097] like Figure 6 As shown, Figure 6 The output image of the annotated panoramic segmentation is shown in the attached figure as a grayscale processed image. The image frame is annotated by panoramic segmentation, and the panoramic segmentation coloring result is obtained, which is then superimposed with the original annotated image to obtain the final image for output.
[0098] The present invention also proposes a data annotation device based on a three-dimensional simulation scene, comprising:
[0099] A preprocessing module is used to obtain a three-dimensional simulation scene, extract image frames, and preprocess the image frames to obtain the original scene coloring result; an input module is used to obtain a labeling type and its corresponding coloring strategy, and obtain the target attribute parameters of each pixel of the target in the image frame according to the labeling type; a coloring module is used to color the image frame according to the target attribute parameters and the coloring strategy to obtain a coloring result; wherein the labeling type includes an illumination attribute, and the coloring strategy corresponding to the illumination attribute includes: dividing the time into segments, taking time periods as units, processing the image frames of each time period, labeling the areas within the illumination range, and superimposing the labeling data at the corresponding pixels of the image frames belonging to the same time period to obtain an illumination coloring result; an output module is used to select the corresponding coloring result according to demand and superimpose it with the original scene coloring result to obtain an output image.
[0100] The advantages of the data annotation device based on the three-dimensional simulation scene over the prior art are the same as those of the three-dimensional simulation scene data annotation method, which will not be described in detail here.
[0101] The present invention also proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is read and executed by a processor, the data labeling method based on the three-dimensional simulation scene as described above is implemented.
[0102] The advantages of the computer-readable storage medium over the prior art and the data annotation method of the three-dimensional simulation scene are the same, which will not be described in detail here.
[0103] Although the present invention is disclosed as above, the protection scope of the present invention is not limited thereto. Those skilled in the art may make various changes and modifications without departing from the spirit and scope of the present invention, and these changes and modifications will fall within the protection scope of the present invention.
Claims
1. A data annotation method based on a three-dimensional simulation scene, characterized in that: include: Acquire a three-dimensional simulation scene, extract an image frame, and preprocess the image frame to obtain an original scene coloring result; Acquire a marking type and a corresponding coloring strategy, and acquire a target attribute parameter of each pixel of the target in the image frame according to the marking type; According to the target attribute parameters and the shading strategy, the image frame is shading, and obtaining the shading result includes: shading the image frame using a vertex shader based on the shading strategy to obtain a vertex shading image; using a fragment shader to sample from texture pixels based on texture coordinates to obtain texture colors, superimposing the texture colors on the vertex shading image, and obtaining the shading result based on the vertex shading image, the shading strategy and the target attribute parameters, wherein the target attribute parameters include at least one of the following: type parameters, speed parameters, acceleration parameters, depth parameters, height parameters and color parameters of the target; wherein the annotation type includes illumination attributes, and the shading strategy corresponding to the illumination attributes includes: segmenting time, taking time periods as units, uniformly processing image frames in each time period, annotating areas within the illumination range, and superimposing the annotation data at corresponding pixels of the image frames belonging to the same time period to obtain an illumination shading result; Select the corresponding shading result according to the needs and superimpose it with the original scene shading result to obtain the output image.
2. The data annotation method based on the three-dimensional simulation scene according to claim 1 is characterized in that: The time is segmented, the image frames in each time period are uniformly processed in units of time periods, the areas within the illumination range are marked, and the marked data at the corresponding pixels of the image frames belonging to the same time period are superimposed to obtain the illumination coloring result, which includes: Segmenting the time period with illumination in the three-dimensional simulation scene, calculating the illumination range of each image frame belonging to the same time period, and rendering the area within the illumination range into a first color; Counting the number of times each pixel of each image frame belonging to the same time period is rendered as the first color; A heat map is calculated according to the number of times, and the heat map is superimposed back on the first image frame of each time period, and rendered to obtain the lighting shading result.
3. The data annotation method based on the three-dimensional simulation scene according to claim 1 is characterized in that: The step of using a vertex shader to perform shading based on the shading strategy to color the image frame and obtain a vertex shaded image comprises: The vertex shader is used to shade the target according to the scene information of the image frame to obtain the vertex shaded image, wherein the scene information includes the viewpoint position and the texture normal.
4. The data annotation method based on the three-dimensional simulation scene according to claim 1 is characterized in that: When the annotation type is a panoramic segmentation attribute, the step of using the fragment shader to obtain a texture color and superimposing the texture color onto the vertex shaded image further includes: Assigning a category and an instance ID to each pixel in the image frame based on the object tags in the image frame, assigning a different color to each object, and generating a panoptic segmentation and coloring result; When the annotation type is a depth attribute, the step of using the fragment shader to obtain a texture color and superimposing the texture color onto the vertex shaded image further includes: Calculating the distance between the pixel point in the image frame and the viewpoint, normalizing the distance and generating a depth annotation coloring result; When the annotation type is a speed attribute, the step of using the fragment shader to obtain a texture color and superimposing the texture color onto the vertex shaded image further includes: Based on the speed of the target in the image frame, marking the object in the image frame, coloring it based on the speed scalar, and generating a speed-labeled coloring result; When the annotation type is a height attribute, the step of using the fragment shader to obtain a texture color and superimposing the texture color onto the vertex shaded image further includes: Coloring is performed based on the heights in the three-dimensional simulation scene corresponding to the pixels in the image frame to generate a height-labeled coloring result.
5. The data annotation method based on three-dimensional simulation scene according to claim 4 is characterized in that: The output image includes: The structured annotation data includes the category, instance ID, distance from the viewpoint, height from the ground, illumination time, and speed, acceleration, position, and direction angle of each pixel in the image frame.
6. The data annotation method based on the three-dimensional simulation scene according to claim 2 is characterized in that: Before calculating the illumination range of each image frame belonging to the same time period, the method further includes: The original scene rendering result and the first image frame of each time period are stored in a temporary queue.
7. A data annotation device based on a three-dimensional simulation scene, characterized in that: include: A preprocessing module, which is used to obtain a three-dimensional simulation scene, extract an image frame, and preprocess the image frame to obtain an original scene coloring result; An input module, which is used to obtain a marking type and a corresponding coloring strategy, and obtain a target attribute parameter of each pixel of the target in the image frame according to the marking type; A shading module, which is used to color the image frame according to the target attribute parameters and the shading strategy, and obtain the shading result, including: coloring the image frame using a vertex shader based on the shading strategy to obtain a vertex shading image; using a fragment shader to sample from texture pixels based on texture coordinates to obtain texture colors, superimposing the texture colors on the vertex shading image, and obtaining the shading result based on the vertex shading image, the shading strategy and the target attribute parameters, wherein the target attribute parameters include at least one of the following: type parameters, speed parameters, acceleration parameters, depth parameters, height parameters and color parameters of the target; wherein the annotation type includes illumination attributes, and the shading strategy corresponding to the illumination attributes includes: segmenting time, taking time periods as units, uniformly processing image frames in each time period, annotating areas within the illumination range, and superimposing the annotation data at corresponding pixels of the image frames belonging to the same time period to obtain illumination shading results; The output module is used to select the corresponding shading result according to the requirements and superimpose it with the original scene shading result to obtain the output image.
8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is read and executed by a processor, the data labeling method based on a three-dimensional simulation scene as described in any one of claims 1 to 6 is implemented.
