Box annotation image expansion method, device and computer-readable storage medium

By performing image segmentation and texture mapping on the box-shaped labeled images, and using a virtual engine and camera to generate a large number of labeled images, the problem of time-consuming and labor-intensive traditional manual labeling methods is solved, and the box-shaped labeled images are expanded and the cost is reduced.

CN112926614BActive Publication Date: 2024-11-29SF TECH CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN201911244765.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2019-12-06
Publication Date
2024-11-29
Estimated Expiration
2039-12-06

AI Technical Summary

Technical Problem

In existing technologies, acquiring box-shaped labeled images requires a lot of manpower and resources, which is difficult and costly. Traditional manual labeling methods are difficult to meet the needs of computer vision for large amounts of labeled data.

Method used

By segmenting the original box-shaped labeled image, multiple single-sided images of the box are obtained. Based on these single-sided images, texture mapping and photography are performed on the 3D model to generate multiple target box-shaped labeled images. The images are then augmented using a virtual engine and a virtual camera.

Benefits of technology

Based on the limited original annotation data, the number of box-shaped annotation images was expanded, reducing the difficulty and cost of obtaining box-shaped annotation images.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112926614B_ABST
    Figure CN112926614B_ABST
Patent Text Reader

Abstract

The embodiment of the present application discloses a box annotation image expansion method, device and computer-readable storage medium, the box annotation image expansion method comprising: obtaining an original box annotation image, wherein the original box annotation image includes annotation information; performing image segmentation on the visible surface of the box in the original box annotation image based on the annotation information to obtain multiple box single-side images; performing texture mapping on each plane of the first box three-dimensional model based on the multiple box single-side images to obtain multiple second box three-dimensional models; taking pictures of the multiple second box three-dimensional models respectively to obtain multiple target box annotation images. The present application realizes the expansion of the number of box annotation images based on a limited number of original box annotation images, which can reduce the difficulty of obtaining box annotation images and reduce the cost of obtaining box annotation images.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a box annotation image expansion method, device and computer-readable storage medium. Background Art

[0002] In the current computer vision field, in order to train and optimize machine deep learning algorithms, a large amount of image and video sample data is required to train and learn the training model. The preparation of a large amount of image training data depends on manual labeling, which requires a lot of manpower investment.

[0003] With the continuous application of machine learning technology in the field of computer vision, the demand for labeled data is increasing. The traditional purely manual data labeling method is difficult to meet the needs of computer vision for a large number of labeled data samples due to its inherent characteristics: limited human energy and time-consuming and labor-intensive. The box measurement based on the software version requires the detection of the key points of the box, and then the volume measurement of the box is achieved based on the coordinates of the key points in three dimensions. However, the detection of box key points based on deep learning requires a large number of labeled box color images. The current existing solutions for key point detection technology based on deep learning methods include: (1) a two-step method, which detects the box based on the detection network and then detects the key points based on the detected box; (2) a one-step method, which directly performs key regression based on the color image. Therefore, it is very difficult to obtain box labeled images.

[0004] Therefore, the use of traditional box labeling solutions requires a lot of manpower and resources, such as operators or hardware resources. In the long run, it is difficult to obtain box labeling images and the cost of obtaining box labeling images is high. Summary of the invention

[0005] The embodiments of the present application provide a box annotation image expansion method, device, and computer-readable storage medium, which can reduce the difficulty of obtaining box annotation images, thereby reducing the cost of obtaining box annotation images.

[0006] In a first aspect, the present application provides a box annotation image expansion method, the box annotation image expansion method comprising:

[0007] Acquire an original box-annotated image, wherein the original box-annotated image includes annotation information;

[0008] Based on the annotation information, the visible surface of the box in the original box annotation image is segmented to obtain multiple single-side images of the box;

[0009] Performing texture mapping on each plane of the first box three-dimensional model based on the multiple box single-side images to obtain multiple second box three-dimensional models;

[0010] Photograph the plurality of second box three-dimensional models respectively to obtain a plurality of target box labeled images.

[0011] The box is a cuboid, and the image segmentation is performed on the visible surface of the box in the original box annotated image based on the annotation information to obtain multiple single-side images of the box, including:

[0012] Determine the edges of the visible surface of the box based on the annotation information;

[0013] The visible surface of the box is segmented based on the edges of the visible surface of the box to obtain a plurality of single-side images of the box.

[0014] The first box model is a cuboid model, and the texture mapping is performed on each plane of the first box three-dimensional model based on the multiple single-side images of the boxes to obtain multiple second box three-dimensional models, including:

[0015] Obtaining the dimensions of three main edges, wherein the three main edges are three edges on the visible surface of the box body that intersect at one point;

[0016] Performing deformation operations on the multiple box single-side images based on the sizes of the three main edges to obtain multiple rectangular box single-side images;

[0017] Based on the multiple rectangular box single-side images, texture mapping is performed on each plane of the first box three-dimensional model to obtain multiple second box three-dimensional models.

[0018] The method of performing texture mapping on each plane of the first box three-dimensional model based on the multiple box single-side images to obtain multiple second box three-dimensional models includes:

[0019] Constructing a first box three-dimensional model in a virtual scene through a virtual engine;

[0020] Based on a plurality of single-side images of the boxes, texture mapping is performed on each plane of the first box three-dimensional model to obtain a plurality of second box three-dimensional models.

[0021] The photographing of the plurality of second box three-dimensional models respectively to obtain a plurality of target box labeled images includes:

[0022] Establish a virtual camera in the virtual scene;

[0023] The virtual camera is controlled to move along a preset motion trajectory, and photographs the second box three-dimensional model at a preset photographing frequency to obtain a plurality of target box annotated images.

[0024] The photographing of the plurality of second box three-dimensional models respectively to obtain a plurality of target box labeled images includes:

[0025] Constructing a noise three-dimensional model around the second box three-dimensional model, wherein the noise three-dimensional model includes at least one of a sphere, a cone and a cylinder;

[0026] The second box three-dimensional model and the noise three-dimensional model are photographed to obtain a plurality of target box annotated images.

[0027] Among them, based on the annotation information, the visible surface of the box in the original box annotation image is segmented to obtain multiple single-side images of the box, including:

[0028] Extract the visible surface of the box from the original box annotation image based on the annotation information;

[0029] Based on the annotation information, the visible surface of the box in the original box annotation image is segmented to obtain multiple single-side images of the box.

[0030] In a second aspect, the present application provides a box body annotation image expansion device, the box body annotation image expansion device comprising:

[0031] An acquisition module, used to acquire an original box annotation image, wherein the original box annotation image includes annotation information;

[0032] An image segmentation module is used to perform image segmentation on the visible surface of the box in the original box annotation image based on the annotation information to obtain multiple single-side images of the box;

[0033] A texture mapping module performs texture mapping on each plane of the first box 3D model based on the multiple box single-side images to obtain multiple second box 3D models;

[0034] The photographing module is used to respectively photograph the plurality of second box three-dimensional models to obtain a plurality of target box annotated images.

[0035] Wherein, the box is a cuboid, and the image segmentation module is further used for:

[0036] Determine the edges of the visible surface of the box based on the annotation information;

[0037] The visible surface of the box is segmented based on the edges of the visible surface of the box to obtain a plurality of single-side images of the box.

[0038] Wherein, the first box model is a cuboid model, and the texture mapping module is further used for:

[0039] Obtaining the dimensions of three main edges, wherein the three main edges are three edges on the visible surface of the box body that intersect at one point;

[0040] Performing deformation operations on the multiple box single-side images based on the sizes of the three main edges to obtain multiple rectangular box single-side images;

[0041] Based on the multiple rectangular box single-side images, texture mapping is performed on each plane of the first box three-dimensional model to obtain multiple second box three-dimensional models.

[0042] Wherein, the texture mapping module is also used for:

[0043] Constructing a first box three-dimensional model in a virtual scene through a virtual engine;

[0044] Based on a plurality of single-side images of the boxes, texture mapping is performed on each plane of the first box three-dimensional model to obtain a plurality of second box three-dimensional models.

[0045] Wherein, the camera module is also used for:

[0046] Establish a virtual camera in the virtual scene;

[0047] The virtual camera is controlled to move along a preset motion trajectory, and photographs the second box three-dimensional model at a preset photographing frequency to obtain a plurality of target box annotated images.

[0048] Wherein, the camera module is also used for:

[0049] Constructing a noise three-dimensional model around the second box three-dimensional model, wherein the noise three-dimensional model includes at least one of a sphere, a cone and a cylinder;

[0050] The second box three-dimensional model and the noise three-dimensional model are photographed to obtain a plurality of target box annotated images.

[0051] Wherein, the acquisition module is also used for:

[0052] Get the original image of the box taken by the user;

[0053] The key points of the original image are annotated to obtain the original box annotated image.

[0054] Wherein, the image segmentation module is also used for:

[0055] Extract the visible surface of the box from the original box annotation image based on the annotation information;

[0056] Based on the annotation information, the visible surface of the box in the original box annotation image is segmented to obtain multiple single-side images of the box.

[0057] In a third aspect, the present application provides a box body annotation image expansion device, the box body annotation image expansion device comprising:

[0058] one or more processors;

[0059] Memory; and

[0060] One or more application programs, wherein the one or more application programs are stored in the memory and are configured to be executed by the processor to implement the box annotation image expansion method according to any one of the first aspects.

[0061] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is loaded by a processor to execute the steps in the box annotation image expansion method described in any one of the first aspects.

[0062] The embodiment of the present application provides a method for expanding a box annotation image, which includes: obtaining an original box annotation image, wherein the original box annotation image includes annotation information; performing image segmentation on the visible surface of the box in the original box annotation image based on the annotation information to obtain multiple box single-side images; performing texture mapping on each plane of the first box three-dimensional model based on the multiple box single-side images to obtain multiple second box three-dimensional models; and taking photos of the multiple second box three-dimensional models respectively to obtain multiple target box annotation images. The present application obtains multiple box single-side images by performing image segmentation on the original box annotation image, and performs different forms of texture mapping on each plane of the first box three-dimensional model through the multiple box single-side images to obtain multiple second box three-dimensional models, and then obtains multiple target box annotation images by taking photos of the multiple second box three-dimensional models, thereby realizing the expansion of the number of box annotation images on the basis of a limited number of original box annotation images, that is, a large amount of annotation data can be generated based on a limited number of original annotation data sets, which can reduce the difficulty of obtaining box annotation images and reduce the cost of obtaining box annotation images. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0064] Figure 1 This is a schematic diagram of an embodiment of a method for expanding a box annotation image provided by an embodiment of the present application;

[0065] Figure 2 yes Figure 1 A schematic flow chart of a specific embodiment of S13;

[0066] Figure 3It is a schematic diagram of the structure of an embodiment of a box body annotation image expansion device provided in an embodiment of the present application;

[0067] Figure 4 It is a schematic diagram of the structure of another embodiment of the box labeling image expansion device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0068] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0069] In the description of the present application, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, which are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the said features. In the description of the present application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0070] In this application, the word "exemplary" is used to mean "used as an example, illustration, or description." Any embodiment described in this application as "exemplary" is not necessarily to be construed as being preferred or advantageous over other embodiments. The following description is given to enable any technician in the field to implement and use the present application. In the following description, details are listed for the purpose of explanation. It should be understood that a person of ordinary skill in the art can recognize that the present application can be implemented without using these specific details. In other instances, well-known structures and processes will not be elaborated in detail to avoid obscuring the description of the present application with unnecessary details. Therefore, the present application is not intended to be limited to the embodiments shown, but is consistent with the widest scope consistent with the principles and features disclosed in the present application.

[0071] The embodiments of the present application provide a box annotation image expansion method, device, and computer-readable storage medium, which are described in detail below.

[0072] First, the present application provides a box annotation image expansion method, which includes: obtaining an original box annotation image, wherein the original box annotation image includes annotation information; performing image segmentation on the visible surface of the box in the original box annotation image based on the annotation information to obtain multiple box single-side images; performing texture mapping on each plane of a first box three-dimensional model based on the multiple box single-side images to obtain multiple second box three-dimensional models; and taking pictures of the multiple second box three-dimensional models respectively to obtain multiple target box annotation images.

[0073] like Figure 1 As shown, Figure 1 : is a schematic diagram of an embodiment of a method for expanding a box body annotation image provided in an embodiment of the present application. The method for expanding a box body annotation image includes:

[0074] S11. Acquire an original box-annotated image, wherein the original box-annotated image includes annotation information.

[0075] In the embodiment of the present application, the annotation information in the original box annotation image may include box key point information. The box key point information may include coordinate data of the box key point and connection line data between the box key points. The box key point may be a box vertex.

[0076] In a specific embodiment, the original image of the box taken by the user is obtained; the key points of the original image of the box are annotated to obtain the original annotated image of the box. For example, the original image of the box obtained by the courier taking a photo of the box with a camera is obtained. After obtaining the original image of the box taken by the user, the key points of the original image of the box can be annotated by manual annotation or automatic machine annotation. Manual annotation has a higher accuracy rate. When the number of original images of the box is small, the accuracy of image annotation can be improved by manual annotation. Automatic machine annotation has a higher efficiency. When the number of original images of the box is large, the efficiency of image annotation can be improved by automatic machine annotation. Automatic machine annotation can establish a detection network model based on posture detection technology, and annotate the original image through the detection network model. Among them, for example, the technology based on posture detection can be a technology based on openpose, a technology based on DeepCut, and a technology based on RMPE (Regional Multi-person Pose Estimation). The method of image annotation is determined according to the specific situation, and this application does not limit this.

[0077] S12: performing image segmentation on the visible surface of the box in the original box annotated image based on the annotation information to obtain a plurality of single-surface images of the box.

[0078] In an embodiment of the present application, the visible surface of the box is extracted from the original box annotated image based on the annotation information; the visible surface of the box in the original box annotated image is segmented based on the annotation information to obtain a plurality of single-side images of the box. The visible surface of the box is the surface of the box facing the observation position. When photographing an object, there will definitely be a part of the surface that is visible to the camera and a part of the surface that is invisible to the camera. The visible part is determined by the position of the box and the position of the camera. And the original box annotated image. Specifically, based on the annotation information, feature points are extracted using a scale-invariant feature transform or a directional gradient histogram method to extract the visible surface of the box. After extracting the visible surface of the box, the visible surface of the box in the original box annotated image is segmented to obtain a plurality of single-side images of the box.

[0079] Obviously, for a cuboid, the cuboid has a total of 6 faces. When there is only one camera, the visible faces of the box can be 1, 2 or 3 single faces. When the visible faces of the box are 1 or 2 single faces, the single faces are all rectangular, and no deformation operation is required. Texture mapping can be performed directly on each plane of the first box 3D model. When the visible faces of the box are 3 single faces, the 3 single faces are all flat quadrilaterals, and deformation operation is required to perform texture mapping on each plane of the first box 3D model.

[0080] In a specific embodiment, the edges of the visible surface of the box are determined based on the annotation information. The visible surface of the box is segmented based on the edges of the visible surface of the box to obtain multiple single-sided images of the box. Since the marking information includes the coordinate data of the key points of the box and the connecting line data between the key points of the box, the coordinates and sizes of the edges of the visible surface of the box are determined according to the marking information. Among them, image segmentation can be performed by image edge segmentation. An important way to segment images is through edge detection, that is, detecting the place where the gray level or structure has a sudden change, indicating the end of one area and the beginning of another area. This discontinuity is called an edge. Different images have different grayscales, and there are generally obvious edges at the boundaries. This feature can be used to segment the image. Obviously, there are 8 edges of the visible surface of the box in the rectangular box, forming a total of 3 single-sided images of the box. Based on the edges of the visible surface of the box, the visible surface of the box is segmented to obtain 3 single-sided images of the box. Therefore, for 2, 3 or more original box annotated images, 3 times the single-sided images of the box can be obtained.

[0081] S13. Performing texture mapping on each plane of the first box 3D model based on the multiple single-side images of the boxes to obtain multiple second box 3D models.

[0082] In the embodiment of the present application, a first box three-dimensional model is constructed in a virtual scene by a virtual engine. Specifically, a first box three-dimensional model is constructed in a virtual scene by a virtual engine. Among them, the virtual engine can be Unity3D, 3dmax and Adobe Photoshop, etc. Among them, Unity3D is a multi-platform comprehensive game development tool developed by Unity Technologies that allows players to easily create interactive content such as three-dimensional video games, architectural visualization, real-time three-dimensional animation, etc. It is a fully integrated professional game engine. 3D Studio Max, often referred to as 3d Max or 3ds MAX, is a three-dimensional animation rendering and production software based on PC system developed by Discreet (later merged by Autodesk). Its predecessor is the 3D Studio series software based on the DOS operating system. In terms of application scope, it is widely used in advertising, film and television, industrial design, architectural design, three-dimensional animation, multimedia production, games, and engineering visualization. AdobePhotoshop, referred to as "PS", is an image processing software developed and released by Adobe Systems. Photoshop mainly processes digital images composed of pixels. Using its numerous editing and drawing tools, image editing can be effectively performed. PS has many functions, covering images, graphics, text, video, publishing and other aspects.

[0083] In a specific embodiment, the first box three-dimensional model is a cuboid. The position coordinates and size of the box are randomly calculated within a configurable range in the virtual scene. The first box three-dimensional model is generated according to the position coordinates and size of the box. A grid can be formed based on triangles, quadrilaterals or other simple convex polygons to construct the first box three-dimensional model. In other embodiments, the first box three-dimensional model can be a cube.

[0084] In an embodiment of the present application, after constructing a first box 3D model in a virtual scene through a virtual engine, texture mapping is performed on each plane of the first box 3D model based on multiple rectangular box single-sided images to obtain multiple second box 3D models. Texture mapping is to paste a planar, two-dimensional texture image onto the geometric surface of an object. When the planar, two-dimensional image and the geometric surface of the object are inconsistent in size, the pixel value of the texture image can be reasonably mapped to the surface of the object through a flexible mapping mechanism, namely, an interpolation function.

[0085] When the size of the first box three-dimensional model is changed, a large number of first box three-dimensional models of different sizes can be obtained, thereby increasing a large number of second box three-dimensional models, and ultimately increasing a large number of box target annotated images, which can reduce the difficulty of obtaining box annotated images and reduce the cost of obtaining box annotated images.

[0086] like Figure 2 As shown, Figure 2 yes Figure 1 In a specific embodiment, texture mapping is performed on each plane of the first box 3D model based on multiple rectangular box single-side images to obtain multiple second box 3D models, which may include:

[0087] S131. Obtain the dimensions of three main edges, where the three main edges are three edges on the visible surface of the box that intersect at one point.

[0088] Specifically, the sizes of the three main edges are obtained according to the coordinate data of the box vertices and the connection line data between the box vertices in the marking information.

[0089] S132: Perform deformation operations on the multiple box single-side images based on the sizes of the three main edges to obtain multiple rectangular box single-side images.

[0090] Specifically, the deformation operation may be affine transformation, enlargement or reduction, etc., which is not limited in this application. For a rectangular box, the three single faces on the visible surface of the box are parallelograms, so an affine transformation operation needs to be performed on the parallelogram to obtain a single-face image of the rectangular box.

[0091] S133: Perform texture mapping on each plane of the first box 3D model based on the multiple rectangular box single-side images to obtain multiple second box 3D models.

[0092] Specifically, multiple rectangular box single-sided images are arranged and combined to obtain multiple mapping methods. By performing texture mapping on each plane of the first box three-dimensional model according to multiple mapping methods, multiple second box three-dimensional models can be obtained. For example, there are 2 original box annotation images, and each original box annotation image is segmented and deformed to obtain 3 rectangular box single-sided images, so a total of 6 rectangular box single-sided images are obtained. The first box three-dimensional model has 6 faces, and there are 720 mapping methods according to the arrangement and combination, so that 720 box models can be obtained. Therefore, through texture mapping, the limited original box annotation images can be expanded, providing a large number of box annotation images, which can reduce the difficulty of obtaining box annotation images and reduce the cost of obtaining box annotation images.

[0093] S14: photographing the plurality of second box three-dimensional models respectively to obtain a plurality of target box labeled images.

[0094] In an embodiment of the present application, a virtual camera is established in a virtual scene; the virtual camera is controlled to move along a preset motion trajectory, and a second box three-dimensional model is photographed at a preset frequency to obtain a plurality of target box labeled images. The virtual camera can be provided by a labeling data generation platform (NDDS-Deep learning Dataset Synthesizer) designed by Nvidia. After the virtual camera is established in the virtual scene, the virtual camera can record the annotation of the content photographed by the virtual camera by receiving the control command. Preferably, the virtual camera is controlled to take pictures and output at least one of key point information, image segmentation information, box boundary information, depth data information and box type.

[0095] In a specific implementation, a virtual camera is established in a virtual scene, and parameter information such as the position, posture, sight line, and field of view of the virtual camera is set. Control information is input into the virtual camera, and the control information includes a preset motion trajectory and a preset photographing frequency, so as to control the virtual camera to photograph the second box 3D model to obtain a plurality of target box labeled images. The virtual camera photographs the second box 3D model from a plurality of different viewing angles, and a large number of target box labeled images can be obtained from each second box 3D model, which can reduce the difficulty of obtaining box labeled images and reduce the cost of obtaining box labeled images.

[0096] Furthermore, a noise three-dimensional model is constructed around the second box three-dimensional model, and the noise three-dimensional model includes at least one of a sphere, a cone and a cylinder. The second box three-dimensional model and the noise three-dimensional model are photographed to obtain a plurality of target box labeled images. Specifically, the virtual camera is controlled to move along a preset motion trajectory, and the second box three-dimensional model and the noise three-dimensional model are photographed at a preset frequency to obtain a plurality of target box labeled images. Image noise refers to unnecessary or redundant interference information present in image data. Various factors in the image that prevent people from receiving its information can be called image noise. Preferably, a noise three-dimensional model is randomly constructed around the second box three-dimensional model. Specifically, the noise three-dimensional model is constructed by a virtual engine, and the specific construction process is the same as the construction process of the first box three-dimensional model, which will not be repeated here.

[0097] Obviously, for each second box three-dimensional model, when the virtual camera takes a picture at a single viewing angle, a noise three-dimensional model is randomly generated around the second box three-dimensional model, and the virtual camera can obtain multiple target box labeled images. In the case of a noise-free image model, the virtual camera can only obtain one target box labeled image at a single viewing angle. In the case of a noisy image model, the virtual camera can obtain multiple target box labeled images at a single viewing angle. Therefore, the present application can obtain an exponentially increased number of target box labeled images, which can reduce the difficulty of obtaining box labeled images and reduce the cost of obtaining box labeled images.

[0098] Therefore, the present application can, in the case of limited original box annotation images, expand the box annotation image by means of multiple box single-side image texture mapping, changing the size of the first box three-dimensional model, adding noise to the second box three-dimensional model, etc., so as to obtain an exponentially increased target box annotation image, which can reduce the difficulty of obtaining the box annotation image and reduce the cost of obtaining the box annotation image.

[0099] The embodiment of the present application provides a method for expanding a box annotation image, which includes: obtaining an original box annotation image, wherein the original box annotation image includes annotation information; performing image segmentation on the visible surface of the box in the original box annotation image based on the annotation information to obtain multiple box single-side images; performing texture mapping on each plane of a first box three-dimensional model based on the multiple box single-side images to obtain multiple second box three-dimensional models; and taking photos of the multiple second box three-dimensional models respectively to obtain multiple target box annotation images. The present application obtains multiple box single-side images by performing image segmentation on the original box annotation image, and performs different forms of texture mapping on each plane of the first box three-dimensional model through the multiple box single-side images to obtain multiple second box three-dimensional models, and then obtains multiple target box annotation images by taking photos of the multiple second box three-dimensional models, thereby realizing the expansion of the number of box annotation images on the basis of a limited number of original box annotation images, that is, a large amount of annotation data can be generated based on a limited number of original annotation data sets, which can reduce the difficulty of obtaining box annotation images and reduce the cost of obtaining box annotation images.

[0100] In order to better implement the box annotation image expansion method in the embodiment of the present application, based on the box annotation image expansion method, the embodiment of the present application also provides a box annotation image expansion device, such as Figure 3 As shown, Figure 3 20 is a schematic diagram of the structure of an embodiment of a box body annotation image expansion device provided in an embodiment of the present application, and the box body annotation image expansion device includes an acquisition module 201, an image segmentation module 202, a texture mapping module 203 and a camera module 204:

[0101] An acquisition module 201 is used to acquire an original box annotation image, wherein the original box annotation image includes annotation information;

[0102] An image segmentation module 202 is used to perform image segmentation on the visible surface of the box in the original box annotation image based on the annotation information to obtain multiple single-side images of the box;

[0103] The texture mapping module 203 performs texture mapping on each plane of the first box 3D model based on the multiple single-side images of the boxes to obtain multiple second box 3D models;

[0104] The photographing module 204 is used to photograph the plurality of second box three-dimensional models respectively to obtain a plurality of target box annotated images.

[0105] The box is a cuboid, and the image segmentation module 202 is also used for:

[0106] Determine the edges of the visible surface of the box based on the annotation information;

[0107] The visible surface of the box is segmented based on the edges of the visible surface of the box to obtain multiple single-side images of the box.

[0108] Wherein, the first box model is a cuboid model, and the texture mapping module 203 is further used for:

[0109] Get the dimensions of three main edges, where the three main edges are the three edges on the visible surface of the box that intersect at one point;

[0110] Performing deformation operations on the multiple box single-side images based on the sizes of the three main edges to obtain multiple rectangular box single-side images;

[0111] Based on the multiple rectangular box single-side images, texture mapping is performed on each plane of the first box three-dimensional model to obtain multiple second box three-dimensional models.

[0112] The texture mapping module 203 is also used for:

[0113] Constructing a first box three-dimensional model in a virtual scene through a virtual engine;

[0114] Based on the multiple single-side images of the boxes, texture mapping is performed on each plane of the first box three-dimensional model to obtain multiple second box three-dimensional models.

[0115] The camera module 204 is also used for:

[0116] Establish a virtual camera in the virtual scene;

[0117] The virtual camera is controlled to move along a preset motion trajectory, and photographs the second box three-dimensional model at a preset photographing frequency to obtain a plurality of target box annotated images.

[0118] The camera module 204 is also used for:

[0119] Constructing a noise three-dimensional model around the second box three-dimensional model, wherein the noise three-dimensional model includes at least one of a sphere, a cone and a cylinder;

[0120] The second box 3D model and the noise 3D model are photographed to obtain a plurality of target box labeled images.

[0121] The acquisition module 201 is also used for:

[0122] Get the original image of the box taken by the user;

[0123] The key points of the original image are annotated to obtain the original box annotated image.

[0124] Among them, the image segmentation module 202 is also used for:

[0125] Extract the visible surface of the box from the original box annotation image based on the annotation information;

[0126] Based on the annotation information, the visible surface of the box in the original box annotation image is segmented to obtain multiple single-side images of the box.

[0127] The present application obtains an original box annotation image, wherein the original box annotation image includes annotation information; based on the annotation information, image segmentation is performed on the visible surface of the box in the original box annotation image to obtain multiple single-side images of the box; texture mapping is performed on each plane of the first box three-dimensional model based on the multiple single-side images of the box to obtain multiple second box three-dimensional models; and multiple second box three-dimensional models are photographed respectively to obtain multiple target box annotation images. The present application obtains multiple single-side images of the box by performing image segmentation on the original box annotation image, and multiple second box three-dimensional models can be obtained by performing different forms of texture mapping on each plane of the first box three-dimensional model through multiple single-side images of the box, and then multiple target box annotation images can be obtained by photographing the multiple second box three-dimensional models, thereby realizing the expansion of the number of box annotation images on the basis of a limited number of original box annotation images, that is, a large amount of annotation data can be generated based on a limited number of original annotation data sets, which can reduce the difficulty of obtaining box annotation images and reduce the cost of obtaining box annotation images.

[0128] The present application also provides a box label image expansion device. Figure 4 As shown, Figure 4 : is another structural diagram of an embodiment of the box body annotation image expansion device provided in the embodiment of the present application, specifically:

[0129] The box body annotation image expansion device may include one or more processing core processors 301, one or more computer-readable computer-readable storage media memories 302, a power supply 303, an input unit 304 and other components. Those skilled in the art will understand that Figure 3 The structure of the box-labeled image expansion device shown in the figure does not constitute a limitation on the box-labeled image expansion device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. Among them:

[0130] The processor 301 is the control center of the box body labeling image expansion device. It uses various interfaces and lines to connect various parts of the entire box body labeling image expansion device. By running or executing software programs and / or modules stored in the memory 302, and calling data stored in the memory 302, it executes various functions of the box body labeling image expansion device and processes data, thereby monitoring the box body labeling image expansion device as a whole. Optionally, the processor 301 may include one or more processing cores; preferably, the processor 301 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, etc., and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 301.

[0131] The memory 302 can be used to store software programs and modules. The processor 301 executes various functional applications and data processing by running the software programs and modules stored in the memory 302. The memory 302 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the box body labeling image expansion device, etc. In addition, the memory 302 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 302 can also include a memory controller to provide the processor 301 with access to the memory 302.

[0132] The box body labeling image expansion device also includes a power supply 303 for supplying power to each component. Preferably, the power supply 303 can be logically connected to the processor 301 through a power management system, so as to manage charging, discharging, and power consumption through the power management system. The power supply 303 can also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0133] The box body label image expansion device may further include an input unit 304, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0134] Although not shown, the box body annotation image expansion device may also include a display unit, etc., which will not be described in detail here. Specifically in this embodiment, the processor 301 in the box body annotation image expansion device will load the executable file corresponding to the process of one or more application programs into the memory 302 according to the following instructions, and the processor 301 will run the application program stored in the memory 302, thereby realizing various functions, as follows:

[0135] Acquire an original box-annotated image, wherein the original box-annotated image includes annotation information;

[0136] Based on the annotation information, the visible surface of the box in the original box annotation image is segmented to obtain multiple single-side images of the box;

[0137] Performing texture mapping on each plane of the first box three-dimensional model based on the multiple box single-side images to obtain multiple second box three-dimensional models;

[0138] Photograph the plurality of second box three-dimensional models respectively to obtain a plurality of target box labeled images.

[0139] The present application obtains an original box annotation image, wherein the original box annotation image includes annotation information; based on the annotation information, image segmentation is performed on the visible surface of the box in the original box annotation image to obtain multiple single-side images of the box; texture mapping is performed on each plane of the first box three-dimensional model based on the multiple single-side images of the box to obtain multiple second box three-dimensional models; and multiple second box three-dimensional models are photographed respectively to obtain multiple target box annotation images. The present application obtains multiple single-side images of the box by performing image segmentation on the original box annotation image, and multiple second box three-dimensional models can be obtained by performing different forms of texture mapping on each plane of the first box three-dimensional model through multiple single-side images of the box, and then multiple target box annotation images can be obtained by photographing the multiple second box three-dimensional models, thereby realizing the expansion of the number of box annotation images on the basis of a limited number of original box annotation images, that is, a large amount of annotation data can be generated based on a limited number of original annotation data sets, which can reduce the difficulty of obtaining box annotation images and reduce the cost of obtaining box annotation images.

[0140] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0141] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0142] To this end, the present application embodiment provides a computer-readable storage medium, which may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc. A computer program is stored thereon, and the computer program is loaded by a processor to execute the steps in any of the box annotation image expansion methods provided in the present application embodiment. For example, the computer program can be loaded by the processor to execute the following steps:

[0143] Acquire an original box-annotated image, wherein the original box-annotated image includes annotation information;

[0144] Based on the annotation information, the visible surface of the box in the original box annotation image is segmented to obtain multiple single-side images of the box;

[0145] Performing texture mapping on each plane of the first box three-dimensional model based on the multiple box single-side images to obtain multiple second box three-dimensional models;

[0146] Photograph the plurality of second box three-dimensional models respectively to obtain a plurality of target box labeled images.

[0147] The present application obtains an original box annotation image, wherein the original box annotation image includes annotation information; based on the annotation information, image segmentation is performed on the visible surface of the box in the original box annotation image to obtain multiple single-side images of the box; texture mapping is performed on each plane of the first box three-dimensional model based on the multiple single-side images of the box to obtain multiple second box three-dimensional models; and multiple second box three-dimensional models are photographed respectively to obtain multiple target box annotation images. The present application obtains multiple single-side images of the box by performing image segmentation on the original box annotation image, and multiple second box three-dimensional models can be obtained by performing different forms of texture mapping on each plane of the first box three-dimensional model through multiple single-side images of the box, and then multiple target box annotation images can be obtained by photographing the multiple second box three-dimensional models, thereby realizing the expansion of the number of box annotation images on the basis of a limited number of original box annotation images, that is, a large amount of annotation data can be generated based on a limited number of original annotation data sets, which can reduce the difficulty of obtaining box annotation images and reduce the cost of obtaining box annotation images.

[0148] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the detailed description of other embodiments above, and will not be repeated here.

[0149] In specific implementation, the above units or structures can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above units or structures can refer to the previous method embodiments, which will not be repeated here.

[0150] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0151] The above is a detailed introduction to a box annotation image expansion method, device and computer-readable storage medium provided in an embodiment of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A box annotation image expansion method, characterized in that: The box annotation image expansion method comprises: Get the original image of the box taken by the user; Annotating the original box image with key points to obtain an original box annotated image, wherein the original box annotated image includes annotation information; Based on the annotation information, image segmentation is performed on the visible surface of the box in the original box annotation image to obtain multiple single-side images of the box; Based on the multiple single-side images of the boxes, texture mapping is performed on each plane of the first box three-dimensional model to obtain multiple second box three-dimensional models, including: Arrange and combine a plurality of the single-side images of the box to obtain a plurality of mapping modes; Constructing a first box three-dimensional model in a virtual scene by using a virtual engine, and changing the size of the first box three-dimensional model to obtain a plurality of first box three-dimensional models of different sizes; Performing texture mapping on each plane of the first box three-dimensional model according to the plurality of mapping methods to obtain a plurality of second box three-dimensional models, so as to expand the limited original box annotation image through texture mapping; The virtual camera is controlled to take pictures of the plurality of second box three-dimensional models respectively, and the virtual camera records the annotations of the photographed contents to obtain a plurality of target box annotated images.

2. The box annotation image expansion method according to claim 1, characterized in that: The box is a cuboid, and image segmentation is performed on the visible surface of the box in the original box annotation image based on the annotation information to obtain multiple single-side images of the box, including: Determine the edges of the visible surface of the box based on the annotation information; The visible surface of the box is segmented based on the edges of the visible surface of the box to obtain a plurality of single-side images of the box.

3. The box annotation image expansion method according to claim 2, characterized in that: The first box three-dimensional model is a cuboid model, and texture mapping is performed on each plane of the first box three-dimensional model based on the multiple box single-side images to obtain multiple second box three-dimensional models, including: Obtaining the dimensions of three main edges, wherein the three main edges are three edges on the visible surface of the box body that intersect at one point; Performing deformation operations on the multiple box single-side images based on the sizes of the three main edges to obtain multiple rectangular box single-side images; Based on the multiple rectangular box single-side images, texture mapping is performed on each plane of the first box three-dimensional model to obtain multiple second box three-dimensional models.

4. The box annotation image expansion method according to claim 1, characterized in that: The controlling virtual camera to take pictures of the plurality of second box three-dimensional models respectively, and the virtual camera recording the annotations of the photographed contents to obtain a plurality of target box annotated images, comprises: Establishing the virtual camera in a virtual scene; The virtual camera is controlled to move along a preset motion trajectory, and photographs the second box three-dimensional model at a preset photographing frequency to obtain a plurality of target box annotated images.

5. The box annotation image expansion method according to claim 1, characterized in that: The controlling virtual camera to take pictures of the plurality of second box three-dimensional models respectively, and the virtual camera recording the annotations of the photographed contents to obtain a plurality of target box annotated images, comprises: Constructing a noise three-dimensional model around the second box three-dimensional model, wherein the noise three-dimensional model includes at least one of a sphere, a cone and a cylinder; The second box three-dimensional model and the noise three-dimensional model are photographed to obtain a plurality of target box annotated images.

6. The box annotation image expansion method according to claim 1, characterized in that: Based on the annotation information, the visible surface of the box in the original box annotation image is segmented to obtain multiple single-side images of the box, including: Extract the visible surface of the box from the original box annotation image based on the annotation information; Based on the annotation information, the visible surface of the box in the original box annotation image is segmented to obtain multiple single-side images of the box.

7. A box labeling image expansion device, characterized in that: The box body annotation image expansion device comprises: An acquisition module is used to acquire an original box image taken by a user, and to annotate key points of the original box image to obtain an original box annotated image, wherein the original box annotated image includes annotation information; An image segmentation module is used to perform image segmentation on the visible surface of the box in the original box annotation image based on the annotation information to obtain multiple single-side images of the box; The texture mapping module performs texture mapping on each plane of the first box 3D model based on the multiple box single-side images to obtain multiple second box 3D models, including: arranging and combining the multiple box single-side images to obtain multiple mapping methods; constructing the first box 3D model in the virtual scene through the virtual engine, changing the size of the first box 3D model to obtain multiple first box 3D models of different sizes; performing texture mapping on each plane of the first box 3D model according to the multiple mapping methods to obtain multiple second box 3D models, so as to expand the limited original box annotation image through texture mapping; The photographing module is used to control the virtual camera to take photographs of the plurality of second box three-dimensional models respectively, and the virtual camera records the annotations of the photographed contents to obtain a plurality of target box annotated images.

8. A box labeling image expansion device, characterized in that: The box body annotation image expansion device comprises: one or more processors; Memory; and One or more applications, wherein the one or more applications are stored in the memory and are configured to be executed by the processor to implement the box annotation image expansion method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and the computer program is loaded by a processor to execute the steps in the box annotation image expansion method according to any one of claims 1 to 6.

Citation Information

Patent Citations

  • Real-time three-dimensional reconstruction method, device and equipment

    CN110148217A

  • Image data generation method and device

    CN110428388A