Sample image synthesis method and device, equipment and medium

Abstract

The invention is applied to the field of intelligent driving, and relates to a sample image synthesis method and device, equipment and a medium, and the method comprises the steps: obtaining a road region in a background picture and a foreground picture; wherein the foreground picture comprises a target object, and the target object represents an object to be placed on the road area; determining a mapping area in the background picture based on the road area; wherein the foreground picture is to be added into the mapping area; performing Laplacian pyramid fusion processing and Poisson fusion processing on the low-frequency information and the high-frequency information in the foreground picture and the background picture in sequence based on the mapping region, and synthesizing a sample image; wherein the sample image comprises a road area and a target object placed on the road area. According to the invention, the acquisition efficiency of the sample image is improved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent driving technology, and in particular to a method, device, equipment and medium for synthesizing a sample image. Background Art

[0002] Perception is an important part of autonomous driving technology. It uses cameras and other equipment on the vehicle to collect images on the road, and then identifies road elements on the road based on the target detection model to determine the location and category of the road elements. The training of the target detection model is inseparable from a large number of sample images including road elements.

[0003] In related technologies, sample images are collected through cameras, webcams and other acquisition devices. However, some rare road elements, such as triangular warning signs, may not be common on daily roads and are difficult to collect, resulting in a small number of sample images, which in turn affects the effect of the target detection model. Summary of the invention

[0004] One of the objects of the present invention is to provide a method for synthesizing a sample image to solve the problem of low efficiency in obtaining sample images in the prior art; a second object is to provide a device for synthesizing a sample image; a third object is to provide an electronic device; a fourth object is to provide a computer-readable storage medium; and a fifth object is to provide a computer program product.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0006] A method for synthesizing a sample image, comprising:

[0007] Acquire a road area and a foreground image in a background image; wherein the foreground image includes a target object, and the target object represents an object to be placed on the road area;

[0008] Based on the road area, determining a mapping area in the background image; wherein the foreground image is to be added to the mapping area;

[0009] Based on the mapping area, the low-frequency information and the high-frequency information in the foreground image and the background image are sequentially subjected to Laplacian pyramid fusion processing and Poisson fusion processing to synthesize a sample image; wherein the sample image includes a road area and a target object placed on the road area.

[0010] Further, based on the mapping area, the low-frequency information and the high-frequency information in the foreground image and the background image are sequentially subjected to Laplacian pyramid fusion processing and Poisson fusion processing to synthesize a sample image, including:

[0011] Performing a Laplacian pyramid construction on the foreground image to obtain low-frequency information of the foreground image and high-frequency information of the foreground image; and performing a Laplacian pyramid construction on the background image to obtain low-frequency information of the background image and high-frequency information of the background image;

[0012] Performing low-frequency fusion processing on the low-frequency information of the foreground image and the low-frequency information of the background image in the mapping area to obtain fused low-frequency information of the background image;

[0013] Performing Poisson fusion processing on the high-frequency information of the foreground image and the high-frequency information of the background image in the mapping area to obtain the high-frequency information of the fused background image;

[0014] In the mapping area, the low-frequency information of the fused background image and the high-frequency information of the fused background image are merged to synthesize the sample image.

[0015] Further, the low-frequency information of the foreground image and the low-frequency information of the background image are subjected to low-frequency fusion processing in the mapping area to obtain the fused low-frequency information of the background image, including:

[0016] The low-frequency information of the foreground image and the low-frequency information of the background image are weighted averaged in the mapping area to obtain the low-frequency information of the fused background image.

[0017] Further, the high-frequency information of the foreground image and the high-frequency information of the background image are subjected to Poisson fusion processing in the mapping area to obtain the fused high-frequency information of the background image, including:

[0018] Acquire first gradient information of high frequency information of the foreground image and second gradient information of high frequency information of the background image in the mapping area;

[0019] Based on the mapping area, determining boundary conditions;

[0020] Constructing a Poisson equation according to the first gradient information, the second gradient information and the boundary condition;

[0021] The Poisson equation is solved to obtain high-frequency information of the fused background image.

[0022] Furthermore, the road area has location information; based on the road area, determining the map area in the background image includes:

[0023] Determining a target location point in the road area based on the location information of the road area;

[0024] Determine a scaling parameter according to the target location point; wherein the scaling parameter represents a scaling ratio of the target object;

[0025] A mapping area in the background image is determined according to the scaling parameter and the target location point.

[0026] Further, determining a scaling parameter of the foreground image according to the target position point includes:

[0027] The coordinate information of the target location point is input into a preset calculation model, and the scaling parameters are output.

[0028] Further, determining the mapping area in the background image according to the scaling parameter and the target position point includes:

[0029] Acquire initial size information of the target object; wherein the initial size information represents size information of the maximum circumscribed rectangular frame of the target object when the target object is located at the bottom edge of the image;

[0030] Determining the scaled size information of the target object according to the scaling parameter and the initial size information;

[0031] Determining vertex coordinate information according to the scaled size information and the target position point;

[0032] According to the vertex coordinate information, a mapping area in the background image is obtained.

[0033] Further, obtaining a texture area in the background image according to the vertex coordinate information includes:

[0034] If it is determined that the vertex coordinate information is all located within the road area, then the rectangular frame corresponding to the vertex coordinates is determined to be the mapping area.

[0035] Further, the road area in the background image is obtained, including:

[0036] Acquire a background image; and input the background image into a preset detection model to determine a road area in the background image; wherein the road area has location information.

[0037] A sample image synthesis device, comprising:

[0038] An acquisition module, used to acquire a road area and a foreground image in a background image; wherein the foreground image includes a target object, and the target object represents an object to be placed on the road area;

[0039] A determination module, used to determine a mapping area in the background image based on the road area; wherein the foreground image is to be added to the mapping area;

[0040] A synthesis module is used to perform Laplace pyramid fusion processing and Poisson fusion processing on low-frequency information and high-frequency information in the foreground image and the background image in sequence based on the map area to synthesize a sample image; wherein the sample image includes a road area and a target object placed on the road area.

[0041] An electronic device, comprising: a memory, a processor;

[0042] The memory stores computer-executable instructions;

[0043] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the above-mentioned sample image synthesis method.

[0044] A computer-readable storage medium stores computer-executable instructions, which are used to implement a sample image synthesis method when executed by a processor.

[0045] A computer program product comprises a computer program, which implements a method for synthesizing a sample image when executed by a processor.

[0046] Beneficial effects of the present invention:

[0047] (1) The present invention enables an electronic device to dynamically determine a mapping area suitable for fusing a foreground image, taking into account the road area in the background image, and to sequentially perform Laplacian pyramid fusion processing and Poisson fusion processing on the low-frequency information and high-frequency information of the foreground image and the background image in the mapping area, thereby ensuring the overall structure and color balance of the image after the mapping area is fused, and making the edge transition of the mapping area natural. This method does not require actual acquisition, thereby improving the efficiency of obtaining sample images, ensuring the number of sample images, and improving the quality of sample images;

[0048] (2) The present invention makes it unnecessary for electronic devices to manually label target objects. Since the position of the mapping area is known and the mapping area is integrated with the target object, the cost of manual labeling is saved. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A schematic diagram of an application scenario provided by an embodiment of the present invention;

[0050] Figure 2 A flow chart of a method for synthesizing a sample image provided by an embodiment of the present invention;

[0051] Figure 3 A flow chart of a method for synthesizing a sample image provided by an embodiment of the present invention;

[0052] Figure 4 A schematic diagram of a target position point and scaling parameters provided by an embodiment of the present invention;

[0053] Figure 5 A flow chart of a method for synthesizing a sample image provided by an embodiment of the present invention;

[0054] Figure 6 A schematic diagram of the structure of a sample image synthesis device provided by an embodiment of the present invention;

[0055] Figure 7 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The following will describe the embodiments of the present invention with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention, not for limiting the scope of protection of the present invention.

[0057] In related technologies, sample images used for training target detection models are mostly collected on the road using cameras and other equipment on vehicles. However, some rare road elements, such as triangular warning signs, may not be common on daily roads, resulting in a small number of sample images, which in turn affects the training effect of the target detection model and causes inaccurate target detection.

[0058] In view of this, an embodiment of the present invention provides a method for synthesizing a sample image, which utilizes an actually collected foreground image including a target object and fuses it to a background image including a road area, so that the target object in the fused image can be placed on the road area. The fused image can then be used as a sample image for model training without the need for actual collection, thereby improving the efficiency of obtaining sample images and ensuring the number of sample images.

[0059] The execution subject of the embodiment of the present invention may be an electronic device with processing capabilities, such as a server, a computer, a laptop computer, etc., and the embodiment of the present invention is not limited here.

[0060] Figure 1 A schematic diagram of an application scenario provided by an embodiment of the present invention. Figure 1As shown, a foreground image including a target object and a background image including a road area can be fused to generate a sample image.

[0061] In some embodiments, the above-mentioned electronic device can implement the sample image synthesis method provided in the embodiment of the present invention to obtain a large number of sample images, and then can train perception models such as target detection models based on the obtained sample images, thereby improving the training effect of the perception model, and thereby improving the number of recognizable objects and recognition accuracy of the perception model; further, the trained perception model can be deployed in an electronic device with processing capabilities such as an electronic control unit or domain controller on the vehicle, and when the vehicle executes intelligent driving applications, the recognition accuracy of the perception model can be improved, thereby improving the safety of the intelligent driving application.

[0062] Figure 2 A flow chart of a method for synthesizing a sample image provided by an embodiment of the present invention. Figure 1 and Figure 2 As shown, the method includes:

[0063] S101, obtaining a road area in a background image and a foreground image.

[0064] Exemplarily, the background image is used to represent a picture with a road area. The embodiment of the present invention does not limit the content of the background image, as long as the background image includes a clear road area. The foreground image includes a target object, which represents an object to be placed on the road area. For example, the target object can be an element that can be placed on the road, such as a warning triangle or a safety cone. The embodiment of the present invention does not limit this and can be set according to actual needs.

[0065] The above-mentioned obtaining of the road region in the background image indicates determining the position information of the road region in the background image, for example, it can be represented by vertex coordinate information of a detection frame of the road region.

[0066] In some possible implementations, the electronic device may first obtain a background image, and then determine the location information of the road area in the background image based on the target detection process. Alternatively, the electronic device may directly obtain the background image after the target detection process, and then directly obtain the location information of the road area.

[0067] In some possible implementations, the electronic device can communicate with other electronic devices to obtain a background image or a foreground image. The other electronic devices may be, for example, a mobile terminal, a computer, etc.; the electronic device may receive a background image or a foreground image imported by an external device. The external device may be, for example, a USB flash drive; or the electronic device may extract a background image or a foreground image from a database.

[0068] S102: Determine a texture area in the background image based on the road area.

[0069] It should be noted that the target object in the foreground image needs to be added to the road area in the background image. Therefore, it is necessary to determine an area where the foreground image can be placed based on the road area, that is, the mapping area, and the foreground image is to be added to the mapping area. It should be understood that the size of the mapping area is consistent with the size of the foreground image. If they are inconsistent, the foreground image can be scaled to make it consistent with the size of the mapping area.

[0070] In some possible implementations, the electronic device may randomly determine an area belonging to the road area as the mapping area based on the location information of the road area; or the electronic device may determine a location point in the road area and determine a scaling parameter when the target object is placed at the location point compared to when the target object is placed at a preset location point, and then determine the mapping area based on the scaling parameter and the location point. In this way, the scaling ratio of the target object can be taken into account, or in other words, the relationship between the location point where the target object is placed and the camera position can be taken into account, so that the position and size of the mapping area are more consistent with the actual ratio of the target object.

[0071] S103 . Based on the mapping area, perform Laplacian pyramid fusion processing and Poisson fusion processing on the low-frequency information and high-frequency information in the foreground image and the background image in sequence to synthesize a sample image.

[0072] Exemplarily, the high-frequency information in the image represents the area where the grayscale value changes dramatically, including the edges, details and noise parts of the image, reflecting the detail information in a small range; the low-frequency information in the image represents the area where the grayscale value changes slowly, including large flat areas and slowly changing color or brightness areas. It should be understood that the foreground image has the low-frequency information and high-frequency information of the foreground image, and the background image has the low-frequency information and high-frequency information of the background image. It should be noted that the embodiments of the present invention do not limit the content corresponding to the low-frequency information and the high-frequency information. The above-mentioned sample image representation is used for the image required for the training of perception models such as target detection, and the sample image includes a road area and a target object placed on the road area.

[0073] In some possible implementations, the electronic device may construct Laplacian pyramids for the foreground image and the background image respectively to obtain the Laplacian pyramid of the foreground image and the Laplacian pyramid of the background image; and then based on the respective Laplacian pyramids, the low-frequency information and high-frequency information of the foreground image, as well as the low-frequency information and high-frequency information of the background image may be obtained respectively, and further, the low-frequency information of the foreground image and the low-frequency information of the background image may be Laplacian pyramid fused in the mapping area, and the high-frequency information of the foreground image and the high-frequency information of the background image may be Poisson fused in the mapping area, to finally obtain a synthesized sample image.

[0074] It should be understood that, as mentioned above, low-frequency information represents areas where grayscale values ​​change slowly, that is, the difference between the grayscale values ​​of the image is small, and the background images of the mapping area are mostly road areas, and the overall grayscale values ​​of the road areas change slowly. Therefore, the use of Laplace pyramid fusion processing can pay more attention to the details of the image and ensure the overall structure and color balance of the image; while high-frequency information represents areas where grayscale values ​​change dramatically, and the human eye is more sensitive to this high-frequency information, such as the edges and contours of the image. Therefore, in the fusion process, it is necessary to ensure the natural transition of the image edges. The use of Poisson fusion can allow the foreground image to be based on the gradient field of the background image of the mapping area to obtain the fused image, so that the edge transition is natural.

[0075] The synthesis method of the sample image provided by the embodiment of the present invention can determine the mapping area based on the road area after obtaining the road area of ​​the background image and the foreground image, and then perform Laplacian pyramid fusion processing on the low-frequency information of the foreground image and the low-frequency information of the background image in the mapping area, and perform Poisson fusion processing on the high-frequency information of the foreground image and the high-frequency information of the background image in the mapping area, and finally obtain a synthesized sample image. In this way, the road area in the background image can be taken into account, and the mapping area suitable for fusing the foreground image can be dynamically determined, and the low-frequency information and high-frequency information of the foreground image and the background image can be Laplacian pyramid fusion processing and Poisson fusion processing can be performed in the mapping area in turn, so as to ensure the overall structure and color balance of the image after the mapping area is fused, and make the edge transition of the mapping area natural. This method does not require actual acquisition, improves the acquisition efficiency of the sample image, and improves the quality of the sample image while ensuring the number of sample images.

[0076] Figure 3 A flow chart of a method for synthesizing a sample image provided by an embodiment of the present invention. Figure 3 As shown, in this embodiment Figure 2 Based on the embodiment shown, the sample image synthesis method provided by the present invention is further described. The method includes:

[0077] S201, obtaining a road area in a background image and a foreground image.

[0078] In some possible implementations, the electronic device may first obtain a background image, and then input the background image into a preset detection model to determine a road area in the background image; wherein the road area has location information. The detection model may be, for example, a YOLOVx series, or a model for image semantic segmentation processing, such as U-net (U-shaped Network), Variable Group Convolutional Neural Network (Vargnet), etc., which is not limited in the embodiments of the present invention.

[0079] For example, Vargnet is used as a detection model. Vargnet is a deep learning model mainly used for tasks such as image segmentation and target detection. It combines encoder-decoder structure, multi-scale feature fusion, bilinear difference, and skip connections to retain feature information between encoding and decoding to help restore high-resolution details, thereby improving segmentation accuracy. At the same time, Vargnet uses variable group convolution, fixes the number of channels for each group, and dynamically changes the number of groups to make the computational intensity more balanced. Therefore, using Vargnet as a detection model has the following advantages:

[0080] a. Lightweight network: Vargnet is a lightweight network suitable for computing in embedded systems, which reduces on-chip / off-chip memory communication and improves model efficiency.

[0081] b. Balanced computational intensity: Vargnet makes the computational intensity of operations in the network more balanced and easier to optimize by fixing the number of channels in each group rather than the number of groups.

[0082] c. Large network capacity: Compared with U-net and others, Vargnet has a larger network capacity, allows a smaller channel number, and helps to reduce the time consumption of off-chip communication.

[0083] It should be noted that the embodiment of the present invention does not limit the loss function used by the detection model. For example, a cross-entropy loss function (Cross-Entropy Loss), also known as logarithmic loss (Log Loss), may be used.

[0084] S202: Determine a target location point in the road area based on the location information of the road area.

[0085] Exemplarily, the target location point represents a location point in the road area. For example, the electronic device can randomly determine a location point in the road area as the target location point based on the location information of the road area, or can select the center point of the road area as the target location point.

[0086] S203: Determine a scaling parameter according to the target location point.

[0087] Exemplarily, the above-mentioned scaling parameter represents the scaling ratio of the target object, and the scaling ratio indicates the scaling ratio of the target object when it is placed at the target position point to the scaling ratio of the target object when it is placed at the preset position point. Figure 4 As shown, Figure 4 A schematic diagram of a target location point and scaling parameters provided by an embodiment of the present invention. Figure 4 Take the target object as a triangular warning sign as an example, where the black area represents the road. When shooting at the same position, the position point where the lower edge of the target object is at the lower edge of the image is used as the preset position point, and the size information of the maximum circumscribed rectangular frame of the target object at this time is recorded. When the target object is placed at the target position point ( Figure 4 When the maximum bounding rectangle of the target object at the target position point is represented by a circle in the figure, there is a scaling ratio between the maximum bounding rectangle of the target object at the target position point and the maximum bounding rectangle at the preset position point, that is, the above-mentioned scaling parameter.

[0088] In some possible implementations, the electronic device may obtain sample position points and sample scaling parameters based on an image including a target object collected in reality, train a computing model based on the sample position points and the sample scaling parameters, obtain a preset computing model, and use the preset computing model to determine the scaling parameters corresponding to the target position point. For example, the electronic device may input the coordinate information of the target position point into a preset computing model and output the scaling parameters. It should be noted that the embodiment of the present invention does not limit the type of the preset computing model, for example, it may be a support vector machine, a convolutional neural network, or other models.

[0089] S204: Determine a mapping area in the background image according to the scaling parameter and the target location point.

[0090] Exemplarily, as mentioned above, the above-mentioned mapping area is used to fuse the foreground image. After determining the scaling parameters according to the target position point, the maximum circumscribed rectangular frame when the above-mentioned target object is placed at the preset position point can be scaled according to the scaling parameters to obtain the scaled rectangular frame, and then a rectangular area can be determined as the mapping area based on the scaled rectangular frame and the target position point.

[0091] Specifically, step S204 may include the following steps:

[0092] S2041. Obtain initial size information of the target object.

[0093] Exemplarily, the initial size information represents the size information of the maximum circumscribed rectangular frame of the target object when the target object is located at the lower edge of the image. The size information may include, for example, the length and width. Figure 4 As shown, the initial size information represents the size information of rectangle A.

[0094] S2042: Determine the scaled size information of the target object according to the scaling parameters and the initial size information.

[0095] Exemplarily, the scaled size information of the target object represents the size information after the initial size information is scaled according to the scaling parameter, referring to Figure 4 As shown, the size information of the target object after scaling represents the size information of rectangle B.

[0096] For example, the initial size information is 1 cm in length and 0.5 cm in width, and the scaling parameter is 0.5. Then the size information of the target object after scaling may be 0.5 cm in length and 0.25 cm in width.

[0097] S2043. Determine vertex coordinate information according to the scaled size information and the target position point.

[0098] Exemplarily, the vertex coordinate information represents the vertex coordinate information of the maximum circumscribed rectangular frame of the target object after scaling. Figure 4 As shown, the vertex coordinate information represents the vertex coordinate information of rectangle B. The scaled target object needs to be placed at the target position point. For example, the target position point can be used as the center point of the scaled target object, or as a vertex, which is not limited in the embodiment of the present invention. Taking the target position point as the center point as an example, the vertex coordinate information can be determined based on the coordinates of the target position point and the scaled size information.

[0099] For example, the scaled size information is represented as length a, width b, and the coordinates of the target position point are (x, y). Then the horizontal coordinate of the vertex in the upper left corner can be (xa / 2, y+b / 2). The calculation method of the coordinates of other vertices is similar and will not be repeated here.

[0100] It should be noted that the image coordinate system in the embodiment of the present invention is fixed, so the coordinates can be uniquely determined in each image.

[0101] S2044. Obtain a mapping area in the background image according to the vertex coordinate information.

[0102] Exemplarily, as mentioned above, after the vertex coordinate information of each vertex is determined, a rectangular area can be determined based on the vertex coordinate information as the mapping area.

[0103] In some possible implementations, the electronic device may further determine whether the vertex coordinate information obtained above is all located in the road area to ensure that the target object is located in the road area. Specifically, if the electronic device determines that the vertex coordinate information is all located in the road area, the rectangular box corresponding to the vertex coordinate is determined to be the mapping area. If it is determined that there is vertex coordinate information that is not located in the road area, step S202 is re-executed to re-determine the target position point until all the vertex coordinates determined are located in the road area.

[0104] In the above manner, the electronic device can determine a mapping area that meets the actual scaling ratio and is within the road area based on the target location point and the scaling parameters, thereby improving the accuracy of the mapping area and making the fused sample image more realistic and accurate.

[0105] S205 , based on the mapping area, perform Laplacian pyramid fusion processing and Poisson fusion processing on the low-frequency information and high-frequency information in the foreground image and the background image in sequence to synthesize a sample image.

[0106] It should be noted that this step can refer to the aforementioned step S103 and will not be repeated here.

[0107] The sample image synthesis method provided by the embodiment of the present invention, after acquiring the background image and the foreground image, determines the target location point through the location information of the road area, and then determines the scaling parameters based on the target location point, and determines a more accurate mapping area within the road area based on the target location point and the scaling parameters, so that the foreground image can be integrated into the road area of ​​the background image to obtain a more accurate and natural sample image. In this way, the electronic device can more accurately and dynamically determine multiple mapping areas, increase the number of sample images, and improve the accuracy and authenticity of the sample images. On the other hand, since the location of the mapping area is known and the mapping area is integrated with the target object, there is no need to manually mark the target object, saving the cost of manual marking.

[0108] Figure 5 A flow chart of a method for synthesizing a sample image provided by an embodiment of the present invention. Figure 5 As shown, in this embodiment Figure 3 Based on the embodiment shown, the sample image synthesis method provided by the present invention is further described. The method includes:

[0109] S301, obtaining a road area in a background image and a foreground image.

[0110] It should be noted that this step is similar to the aforementioned step S201 and will not be repeated here.

[0111] S302: Determine a texture area in the background image based on the road area.

[0112] It should be noted that this step is similar to the aforementioned steps S202-S204 and will not be repeated here.

[0113] S303, constructing a Laplacian pyramid for the foreground image to obtain low-frequency information and high-frequency information of the foreground image; and constructing a Laplacian pyramid for the background image to obtain low-frequency information and high-frequency information of the background image.

[0114] Exemplarily, the electronic device may construct a Laplacian pyramid for the foreground image to obtain a Laplacian pyramid image of the foreground image, and use the bottom layer in the Laplacian pyramid image of the foreground image as the low-frequency information of the foreground image, and then use the Laplacian pyramid image of the foreground image to subtract the bottom layer to obtain the high-frequency information of the foreground image. Similarly, the low-frequency information of the background image and the high-frequency information of the background image may be obtained. It should be noted that the embodiment of the present invention does not limit how to construct the Laplacian pyramid.

[0115] S304: Perform low-frequency fusion processing on the low-frequency information of the foreground image and the low-frequency information of the background image in the mapping area to obtain fused low-frequency information of the background image.

[0116] For example, the map area needs to fuse the background image and the foreground image. Therefore, only the content of the map area of ​​the background image can be fused with the foreground image, and the other areas of the background image remain unchanged. Therefore, the electronic device can only perform low-frequency fusion processing on the low-frequency information of the map area of ​​the background image and the low-frequency information of the foreground image. For example, in the map area, the low-frequency information of the background image and the low-frequency information of the foreground image are weighted averaged to obtain the low-frequency information of the fused background image. Specifically, a weighted fusion formula can be used for the map area, and the fused image of the map area = ×Low frequency information of the foreground image + (1- ) × the low-frequency information of the background image in the texture area to obtain the low-frequency information of the fused background image. It is a preset weight coefficient, which is not limited in the embodiment of the present invention.

[0117] S305 , performing Poisson fusion processing on the high-frequency information of the foreground image and the high-frequency information of the background image in the mapping area to obtain the fused high-frequency information of the background image.

[0118] Exemplarily, as mentioned above, after the Laplacian pyramid is constructed for the foreground image and the background image, the low-frequency information of the foreground image and the low-frequency information of the background image can be fused in the mapping area, so that the low-frequency information is fused. However, for high-frequency information, if only the high-frequency information of the background image and the high-frequency information of the foreground image are weighted averaged, the high-frequency information of the fused image will be unnatural and will cause loss of details. For example, taking the target object in the foreground image as a triangular warning sign as an example, the edge of the triangular warning sign is a brighter color, such as red and yellow, and the middle is hollow, which may be the color of the road. When converted to the image, the transition from the edge to the middle of the triangular warning sign can be regarded as an area where the grayscale value changes dramatically, that is, high-frequency information. For this area, if only the foreground image and the background image are weighted averaged, the color change will be abrupt, and the human eye is sensitive to high-frequency information, resulting in an unnatural and unreal fused image. Therefore, Poisson fusion can be used for fusion processing for high-frequency information, so that the foreground image can be naturally transitioned based on the gradient field of the mapping area of ​​the background image.

[0119] In some possible implementations, step S305 may include the following steps:

[0120] S3051. Obtain first gradient information of high-frequency information of the foreground image and second gradient information of high-frequency information of the background image in the mapping area.

[0121] Exemplarily, the electronic device may calculate gradient information based on a preset edge detection operator, where the gradient information includes a gradient magnitude and a gradient direction. The edge detection operator may be, for example, a Sobel operator or a Prewitt operator, which is not limited in the embodiment of the present invention. It should be noted that the embodiment of the present invention does not limit how to calculate the gradient information.

[0122] S3052: Determine boundary conditions based on the mapping area.

[0123] Exemplarily, the boundary condition may be, for example, a pixel value adjacent to the mapping area in high-frequency information of the background image.

[0124] S3053. Construct a Poisson equation according to the first gradient information, the second gradient information and the boundary conditions.

[0125] For example, the Poisson equation can generally be expressed as: ,in, is the Laplace operator, expressed as: , that is, the Laplace operator can be used to describe the second-order derivative of a function in the x-direction and the y-direction; Represents the pixel value of the fused image at each pixel ; Indicates the change of gradient information of the fused image at each pixel; Represents the second gradient information of each pixel in the high-frequency information of the background image, Represents the first gradient information of each pixel in the high-frequency information of the foreground image.

[0126] S3054. Solve the Poisson equation to obtain high-frequency information of the fused background image.

[0127] Exemplarily, the electronic device may use numerical methods, such as Gauss-Seidel iteration, conjugate gradient method or multigrid method, to solve Poisson's equation to obtain the pixel value of each pixel of the fused image, and further obtain high-frequency information of the fused background image.

[0128] S306 , in the mapping area, combining the low-frequency information of the fused background image and the high-frequency information of the fused background image to synthesize a sample image.

[0129] Exemplarily, the electronic device may perform weighted summation of the low-frequency information of the fused background image and the high-frequency information of the fused background image to synthesize the sample image. For example, the weight coefficient of the low-frequency information and the weight coefficient of the high-frequency information may be set separately, and the low-frequency information of the fused background image and the high-frequency information of the fused background image may be weighted summated using different weight coefficients to synthesize the sample image.

[0130] The synthesis method of the sample image provided by the embodiment of the present invention, after obtaining the road area and the foreground image of the background image and determining the mapping area, performs low-frequency fusion processing in the mapping area for the low-frequency information of the foreground image and the low-frequency information of the background image after the Laplacian pyramid is constructed, and obtains the low-frequency information of the fused background image, and performs Poisson fusion processing in the mapping area for the high-frequency information of the foreground image and the high-frequency information of the background image, and obtains the high-frequency information of the fused background image, and finally synthesizes the sample image in the mapping area. In this way, different fusion methods can be used for processing the high-frequency information and the low-frequency information, so that the overall structure and color of the fused sample image are balanced, and the edge transition of the mapping area is natural, thereby improving the authenticity and naturalness of the sample image.

[0131] Figure 6 A schematic diagram of the structure of a sample image synthesis device provided by an embodiment of the present invention is shown in FIG. Figure 6 As shown, the sample image synthesis device 400 of the embodiment of the present invention includes:

[0132] The acquisition module 401 is used to acquire a road area and a foreground image in a background image; wherein the foreground image includes a target object, and the target object represents an object to be placed on the road area;

[0133] A determination module 402 is used to determine a mapping area in the background image based on the road area; wherein the foreground image is to be added to the mapping area;

[0134] The synthesis module 403 is used to perform Laplace pyramid fusion processing and Poisson fusion processing on the low-frequency information and high-frequency information in the foreground image and the background image in sequence based on the map area to synthesize a sample image; wherein the sample image includes a road area and a target object placed on the road area.

[0135] In some possible implementations, the synthesis module 403 is specifically configured to:

[0136] Performing a Laplacian pyramid construction on the foreground image to obtain low-frequency information of the foreground image and high-frequency information of the foreground image; and performing a Laplacian pyramid construction on the background image to obtain low-frequency information of the background image and high-frequency information of the background image;

[0137] Performing low-frequency fusion processing on the low-frequency information of the foreground image and the low-frequency information of the background image in the mapping area to obtain fused low-frequency information of the background image;

[0138] Performing Poisson fusion processing on the high-frequency information of the foreground image and the high-frequency information of the background image in the mapping area to obtain the high-frequency information of the fused background image;

[0139] In the mapping area, the low-frequency information of the fused background image and the high-frequency information of the fused background image are merged to synthesize the sample image.

[0140] In some possible implementations, the synthesis module 403 is specifically configured to:

[0141] The low-frequency information of the foreground image and the low-frequency information of the background image are weighted averaged in the mapping area to obtain the low-frequency information of the fused background image.

[0142] In some possible implementations, the synthesis module 403 is specifically configured to:

[0143] Acquire first gradient information of high frequency information of the foreground image and second gradient information of high frequency information of the background image in the mapping area;

[0144] Based on the mapping area, determining boundary conditions;

[0145] Constructing a Poisson equation according to the first gradient information, the second gradient information and the boundary condition;

[0146] The Poisson equation is solved to obtain high-frequency information of the fused background image.

[0147] In some possible implementations, the road area has location information; the determination module 402 is specifically configured to:

[0148] Determining a target location point in the road area based on the location information of the road area;

[0149] Determine a scaling parameter according to the target location point; wherein the scaling parameter represents a scaling ratio of the target object;

[0150] A mapping area in the background image is determined according to the scaling parameter and the target location point.

[0151] In some possible implementations, the determining module 402 is specifically configured to:

[0152] The coordinate information of the target location point is input into a preset calculation model, and the scaling parameters are output.

[0153] In some possible implementations, the determining module 402 is specifically configured to:

[0154] Acquire initial size information of the target object; wherein the initial size information represents size information of the maximum circumscribed rectangular frame of the target object when the target object is located at the bottom edge of the image;

[0155] Determining the scaled size information of the target object according to the scaling parameter and the initial size information;

[0156] Determining vertex coordinate information according to the scaled size information and the target position point;

[0157] According to the vertex coordinate information, a mapping area in the background image is obtained.

[0158] In some possible implementations, the determining module 402 is specifically configured to:

[0159] If it is determined that the vertex coordinate information is all located within the road area, then the rectangular frame corresponding to the vertex coordinates is determined to be the mapping area.

[0160] In some possible implementations, the acquisition module 401 is specifically configured to:

[0161] Acquire a background image; and input the background image into a preset detection model to determine a road area in the background image; wherein the road area has location information.

[0162] The device of the embodiment of the present invention can be used to execute the technical solution of any of the above-mentioned method embodiments. Its implementation principle and technical effects are similar and will not be repeated here.

[0163] Figure 7 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. Figure 7 As shown, the electronic device 500 may include: at least one processor 501 and a memory 502 .

[0164] The memory 502 is used to store programs. Specifically, the programs may include program codes, and the program codes include computer-executable instructions.

[0165] The memory 502 may include a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0166] The processor 501 is used to execute the computer-executable instructions stored in the memory 502 to implement the sample image synthesis method described in the aforementioned method embodiment. The processor 501 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiment of the present invention. Specifically, when implementing the sample image synthesis method described in the aforementioned method embodiment, the electronic device may be, for example, an electronic device with processing functions such as a server or a computer.

[0167] Optionally, the electronic device 500 may further include a communication interface 503. In a specific implementation, if the communication interface 503, the memory 502 and the processor 501 are implemented independently, the communication interface 503, the memory 502 and the processor 501 may be interconnected through a bus and communicate with each other. The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc., but it does not mean that there is only one bus or one type of bus.

[0168] Optionally, in a specific implementation, if the communication interface 503, the memory 502 and the processor 501 are integrated on a chip, the communication interface 503, the memory 502 and the processor 501 can communicate through an internal interface.

[0169] The present invention also provides a computer-readable storage medium, in which computer program instructions are stored. When a processor executes the computer program instructions, the above-mentioned method for synthesizing sample images is implemented.

[0170] The present invention also provides a computer program product, including a computer program, which implements the above-mentioned method for synthesizing sample images when executed by a processor.

[0171] The computer-readable storage medium mentioned above may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), erasable programmable read only memory (EPROM), programmable read only memory (PROM), read only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk. The readable storage medium may be any available medium that can be accessed by a general or special purpose computer.

[0172] An exemplary readable storage medium is coupled to the processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be an integral part of the processor. The processor and the readable storage medium can be located in a dedicated integrated circuit. Of course, the processor and the readable storage medium can also exist as discrete components in the sample image synthesis device.

[0173] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0174] The above embodiments are only preferred embodiments for fully illustrating the present invention, and the protection scope of the present invention is not limited thereto. Any equivalent substitution or change made by a person skilled in the art based on the present invention is within the protection scope of the present invention.

Claims

1. A method for synthesizing a sample image, characterized in that: The method comprises: Acquire a road area and a foreground image in a background image; wherein the foreground image includes a target object, and the target object represents an object to be placed on the road area; Based on the road area, determining a mapping area in the background image; wherein the foreground image is to be added to the mapping area; Based on the mapping area, the low-frequency information and the high-frequency information in the foreground image and the background image are sequentially subjected to Laplacian pyramid fusion processing and Poisson fusion processing to synthesize a sample image; wherein the sample image includes a road area and a target object placed on the road area.

2. The method according to claim 1, characterized in that Based on the mapping area, the low-frequency information and the high-frequency information in the foreground image and the background image are sequentially subjected to Laplacian pyramid fusion processing and Poisson fusion processing to synthesize a sample image, including: Performing a Laplacian pyramid construction on the foreground image to obtain low-frequency information of the foreground image and high-frequency information of the foreground image; and performing a Laplacian pyramid construction on the background image to obtain low-frequency information of the background image and high-frequency information of the background image; Performing low-frequency fusion processing on the low-frequency information of the foreground image and the low-frequency information of the background image in the mapping area to obtain fused low-frequency information of the background image; Performing Poisson fusion processing on the high-frequency information of the foreground image and the high-frequency information of the background image in the mapping area to obtain the high-frequency information of the fused background image; In the mapping area, the low-frequency information of the fused background image and the high-frequency information of the fused background image are merged to synthesize the sample image.

3. The method according to claim 2, characterized in that The low-frequency information of the foreground image and the low-frequency information of the background image are subjected to low-frequency fusion processing in the mapping area to obtain the fused low-frequency information of the background image, including: The low-frequency information of the foreground image and the low-frequency information of the background image are weighted averaged in the mapping area to obtain the low-frequency information of the fused background image.

4. The method according to claim 2, characterized in that: The high-frequency information of the foreground image and the high-frequency information of the background image are subjected to Poisson fusion processing in the mapping area to obtain the fused high-frequency information of the background image, including: Acquire first gradient information of high frequency information of the foreground image and second gradient information of high frequency information of the background image in the mapping area; Based on the mapping area, determining boundary conditions; Constructing a Poisson equation according to the first gradient information, the second gradient information and the boundary condition; The Poisson equation is solved to obtain high-frequency information of the fused background image.

5. The method according to claim 1, characterized in that The road area has position information; based on the road area, determining the map area in the background image includes: Determining a target location point in the road area based on the location information of the road area; Determine a scaling parameter according to the target location point; wherein the scaling parameter represents a scaling ratio of the target object; A mapping area in the background image is determined according to the scaling parameter and the target location point.

6. The method according to claim 5, characterized in that Determining a scaling parameter of the foreground image according to the target position point includes: The coordinate information of the target location point is input into a preset calculation model, and the scaling parameters are output.

7. The method according to claim 5, characterized in that Determining a mapping area in the background image according to the scaling parameter and the target position point includes: Acquire initial size information of the target object; wherein the initial size information represents size information of the maximum circumscribed rectangular frame of the target object when the target object is located at the bottom edge of the image; Determining the scaled size information of the target object according to the scaling parameter and the initial size information; Determining vertex coordinate information according to the scaled size information and the target position point; According to the vertex coordinate information, a mapping area in the background image is obtained.

8. The method according to claim 7, characterized in that According to the vertex coordinate information, a mapping area in the background image is obtained, including: If it is determined that the vertex coordinate information is all located within the road area, then the rectangular frame corresponding to the vertex coordinates is determined to be the mapping area.

9. The method according to any one of claims 1 to 8, characterized in that: Get the road area in the background image, including: Acquire a background image; and input the background image into a preset detection model to determine a road area in the background image; wherein the road area has location information.

10. A sample image synthesis device, characterized in that: The device comprises: An acquisition module, used to acquire a road area and a foreground image in a background image; wherein the foreground image includes a target object, and the target object represents an object to be placed on the road area; A determination module, used to determine a mapping area in the background image based on the road area; wherein the foreground image is to be added to the mapping area; A synthesis module is used to perform Laplace pyramid fusion processing and Poisson fusion processing on low-frequency information and high-frequency information in the foreground image and the background image in sequence based on the map area to synthesize a sample image; wherein the sample image includes a road area and a target object placed on the road area.

11. An electronic device, characterized in that: include: Memory, processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 9 when executed by a processor.

13. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 9 when being executed by a processor.

Images