Image processing method and device

Abstract

The present disclosure provides an image processing method and device. The image processing method includes: extracting a first mask image of a target in a scene image and a normal of the target in a camera coordinate system; obtaining a first boundary line segment set of the target in a pixel coordinate system according to the first mask image; mapping the first mask image and each boundary line segment in the first boundary line segment set to a camera plane coordinate system according to the normal and camera preset parameters to generate a second mask image and a second boundary line segment set; in the plane coordinate system, generating a minimum bounding rectangle of the target according to the second mask image and the second boundary line segment set; mapping a preset template image to the minimum bounding rectangle of the target to generate a transformed image; replacing the target in the scene image with the transformed image to generate a virtual image.

Description

Technical Field

[0001] The present disclosure relates to the field of information processing, and in particular to an image processing method and device. Background Art

[0002] At present, image display has been widely used as a means of publicity. By displaying items such as floors and carpets in different decoration scenes, the publicity effect can be effectively improved. For example, by pasting the floor template image to the floor area in the scene image through perspective transformation, the virtual image looks like it was taken in the real scene. Summary of the invention

[0003] The inventors noticed that in the related art, the location of the floor map is selected manually, which has high labor costs and is time-consuming, and the user experience is poor.

[0004] Based on this, the present disclosure provides an image processing solution that can easily and quickly replace a specified target in a scene image, effectively improving the user experience.

[0005] According to a first aspect of an embodiment of the present disclosure, there is provided an image processing method, comprising: extracting a first mask image of a target in a scene image and a normal of the target in a camera coordinate system; acquiring a first boundary line segment set of the target in a pixel coordinate system according to the first mask image; mapping the first mask image and each boundary line segment in the first boundary line segment set to a camera plane coordinate system according to the normal and preset parameters of the camera to generate a second mask image and a second boundary line segment set; in the plane coordinate system, generating a minimum circumscribed rectangle of the target according to the second mask image and the second boundary line segment set; mapping a preset template image to the minimum circumscribed rectangle of the target to generate a transformed image; and replacing the target in the scene image with the transformed image to generate a virtual image.

[0006] In some embodiments, obtaining a first boundary line segment set of the target in a pixel coordinate system based on the first mask image includes: performing straight line detection on the first mask image to generate a first candidate line segment set; deleting the boundary line segments of the scene image in the first candidate line segment set to generate a second candidate line segment set; and deleting the boundary line segments of objects located in the target area in the second candidate line segment set to generate the first boundary line segment set.

[0007] In some embodiments, deleting the boundary line segments of the scene image in the first candidate line segment set includes: determining whether the i-th line segment in the first candidate line segment set satisfies a first condition or a second condition, wherein the first condition is |x1-x2|<β and |x1+x2-W|>αW, the second condition is |y1-y2|<β and |y1+y2-H|>αH, the coordinates of the two endpoints of the i-th line segment are x1, y1 and x1, y2 respectively, the size of the first mask image is H×W, α and β are preset parameters, 1≤i≤n1, n1 is the number of candidate line segments in the first candidate line segment set; if the i-th line segment satisfies the first condition or the second condition, the i-th line segment is deleted from the first candidate line segment set.

[0008] In some embodiments, deleting the boundary line segments of objects located in the target area in the second candidate line segment set includes: determining whether the j-th line segment in the second candidate line segment set satisfies a third condition, wherein the third condition is abs(mean(reg1)-mean(reg2))<γ, mean(reg1) is the mean pixel value of the pixel points in the first area on one side of the j-th line in the minimum circumscribed convex polygon of the first mask image, mean(reg2) is the mean pixel value of the pixel points in the second area on the other side of the j-th line in the minimum circumscribed convex polygon of the first mask image, abs is an absolute value function, γ is a preset parameter, 1≤j≤n2, n2 is the number of candidate line segments in the second candidate line segment set; if the j-th line segment satisfies the third condition, the j-th line segment is deleted from the second candidate line segment set.

[0009] In some embodiments, mapping the first mask image and each boundary line segment in the first boundary line segment set to the camera plane coordinate system according to the normal and camera preset parameters includes: mapping the first coordinate of the foreground pixel point in the first mask image to the camera coordinate system to obtain the corresponding second coordinate according to the normal and camera preset parameters; mapping the second coordinate to the plane coordinate system to obtain the corresponding third coordinate; calculating a first transformation submatrix from the first coordinate to the third coordinate; using the first transformation submatrix, mapping the first mask image to the plane coordinate system to generate a second mask image; using the first transformation submatrix, mapping each line segment in the first boundary line segment set to the plane coordinate system to generate a second boundary line segment set.

[0010] In some embodiments, in the plane coordinate system, generating the minimum bounding rectangle of the target based on the second mask image and the second boundary line segment set includes: selecting the line segment with the largest length in the second boundary line segment set as a reference line segment; calculating the angle θ between the reference line segment and the horizontal direction; rotating the second mask image by an angle θ along a first direction to generate a third mask image, so that the rotated reference line segment is parallel to the horizontal direction; generating a minimum bounding rectangle of the foreground area in the third mask image; rotating the minimum bounding rectangle of the foreground area generated in the third mask image by an angle θ along a second direction opposite to the first direction to obtain the minimum bounding rectangle of the target.

[0011] In some embodiments, mapping a preset template image into the minimum bounding rectangle of the target to generate a transformed image includes: transforming the minimum bounding rectangle of the target into a square; calculating a second transformation submatrix that maps the template image into the square; obtaining a transformation matrix based on the first transformation submatrix and the second transformation submatrix; and mapping the template image into the pixel coordinate system based on the transformation matrix to generate the transformed image.

[0012] In some embodiments, the transformation matrix is ​​the product of the inverse matrix of the first transformation sub-matrix and the second transformation sub-matrix.

[0013] In some embodiments, replacing the target in the scene image with the conversion image includes: when the pixel value of the pixel point in the i-th row and j-th column in the first mask image is greater than 0, replacing the pixel value of the pixel point in the i-th row and j-th column in the scene image with the pixel value of the pixel point in the i-th row and j-th column in the conversion image.

[0014] According to a second aspect of an embodiment of the present disclosure, there is provided an image processing device, comprising: a first processing module, configured to extract a first mask image of a target in a scene image and a normal of the target in a camera coordinate system; a second processing module, configured to obtain a first boundary line segment set of the target in a pixel coordinate system according to the first mask image; a third processing module, configured to map the first mask image and each boundary line segment in the first boundary line segment set to a camera plane coordinate system according to the normal and preset camera parameters to generate a second mask image and a second boundary line segment set; a fourth processing module, configured to generate a minimum bounding rectangle of the target in the plane coordinate system according to the second mask image and the second boundary line segment set; a fifth processing module, configured to map a preset template image to the minimum bounding rectangle of the target to generate a transformed image; and an image updating module, configured to replace the target in the scene image with the transformed image to generate a virtual image.

[0015] According to a third aspect of an embodiment of the present disclosure, there is provided an image processing apparatus, including:

[0016] The memory is configured to store instructions; the processor is coupled to the memory, and the processor is configured to execute the method described in any of the above embodiments based on the instructions stored in the memory.

[0017] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and when the instructions are executed by a processor, the method involved in any of the above embodiments is implemented.

[0018] Other features and advantages of the present disclosure will become apparent from the following detailed description of exemplary embodiments of the present disclosure with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0020] Figure 1 A schematic diagram of a flow chart of an image processing method according to an embodiment of the present disclosure;

[0021] FIG. 2A to FIG. 2C Schematic diagram of a scene image and a mask image according to some embodiments of the present disclosure;

[0022] Figure 3A and Figure 3B is a schematic diagram of a mask image according to some embodiments of the present disclosure;

[0023] Figure 4 is a schematic diagram of a mask image according to another embodiment of the present disclosure;

[0024] Figure 5 is a schematic diagram of a mask image according to another embodiment of the present disclosure;

[0025] Figure 6 is a schematic diagram of a mask image according to another embodiment of the present disclosure;

[0026] Figure 7 is a schematic diagram of a mask image according to another embodiment of the present disclosure;

[0027] Figure 8 A schematic diagram of a template image according to an embodiment of the present disclosure;

[0028] Fig. 9is a schematic diagram of a template image according to another embodiment of the present disclosure;

[0029] Fig.10 A schematic diagram of a virtual image according to an embodiment of the present disclosure;

[0030] Fig.11 A schematic diagram of the structure of an image processing device according to an embodiment of the present disclosure;

[0031] Fig.12 The figure is a schematic diagram of the structure of an image processing device according to another embodiment of the present disclosure. DETAILED DESCRIPTION

[0032] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is by no means intended to limit the present disclosure and its application or use. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0033] Unless specifically stated otherwise, the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure.

[0034] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0035] Technologies, methods, and apparatus known to ordinary technicians in the relevant field may not be discussed in detail, but where appropriate, such technologies, methods, and apparatus should be considered part of the authorization specification.

[0036] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0037] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0038] Figure 1 The following is a flow chart of an image processing method according to an embodiment of the present disclosure. In some embodiments, the following steps of the image processing method are performed by an image processing device.

[0039] In step 101, a first mask image of an object in a scene image and a normal line of the object in a camera coordinate system are extracted.

[0040] In some embodiments, the LiteSeg network is used to extract the scene image I 0 In addition, by extracting the intermediate layer feature f of the LiteSeg network, a global pooling operation is performed after a convolution operation to obtain a feature map with a predetermined size (for example, 1×1×1024), and finally the normal N of the target in the camera coordinate system is obtained through a fully connected layer. For example, the scene image I 0 Zhongwei Figure 2A As shown, the first mask image M1 is Figure 2B shown.

[0041] For example, Figure 2A As shown, the scene image I 0 The target is the floor.

[0042] For example, the normal N can be expressed as:

[0043] N = {a, b, c, d} (1)

[0044] In formula (1), (a, b, c) represents the unit normal of the plane. If the coordinates of a point on the target plane in the camera coordinate system are (x, y, z), then:

[0045] ax+by+cx+d=0 (2)

[0046] It should be noted here that since the LiteSeg network itself is not the inventive point of the present disclosure, it will not be described in detail here.

[0047] In step 102, a first boundary line segment set of the target in a pixel coordinate system is obtained according to the first mask image.

[0048] In some embodiments, straight line detection is first performed on the first mask image to generate a first candidate line segment set.

[0049] For example, by using Hough line detection, n1 boundary line segments in the first mask image M1 are obtained to obtain the first candidate line segment set

[0050] Next, the boundary line segments of the scene image in the first candidate line segment set are deleted to generate a second candidate line segment set.

[0051] For example, determine whether the i-th line segment in the first candidate line segment set L1 satisfies the first condition or the second condition, wherein the first condition is |x1-x2|<β and |x1+x2-W|>αW, the second condition is |y1-y2|<β and |y1+y2-H|>αH, the coordinates of the two endpoints of the i-th line segment are x1, y1 and x1, y2 respectively, the size of the first mask image is H×W, α and β are preset parameters, 1≤i≤n1, and n1 is the number of candidate line segments in the first candidate line segment set. If the i-th line segment satisfies the first condition or the second condition, delete the i-th line segment from the first candidate line segment set.

[0052] That is, the scene image 1 is deleted from the first candidate line segment L1 using the following formulas (3) and (4): 0 To generate the second candidate segment set L 2 .

[0053]

[0054]

[0055] Then, the boundary line segments of the object located in the target area in the second candidate line segment set are deleted to generate a first boundary line segment set.

[0056] For example, determine whether the j-th line segment in the second candidate line segment set satisfies the third condition, where the third condition is abs(mean(reg1)-mean(reg2))<γ, mean(reg1) is the mean pixel value of the pixel points in the first area on one side of the j-th line in the minimum circumscribed convex polygon of the first mask image, mean(reg2) is the mean pixel value of the pixel points in the second area on the other side of the j-th line in the minimum circumscribed convex polygon of the first mask image, abs is the absolute value function, γ is a preset parameter, 1≤j≤n2, n2 is the number of candidate line segments in the second candidate line segment set. If the j-th line segment satisfies the third condition, delete the j-th line segment from the second candidate line segment set. The minimum circumscribed convex polygon of the first mask image is as follows: Figure 2C shown.

[0057] That is, use the following formulas (5) and (6) to select the second candidate line segment set L 2 The boundary line segments of the objects in the target area are deleted to generate a first boundary line segment set L 3 .

[0058]

[0059]

[0060] In step 103, the first mask image and each boundary line segment in the first boundary line segment set are mapped to a camera plane coordinate system according to the normal line and camera preset parameters to generate a second mask image and a second boundary line segment set.

[0061] In some embodiments, firstly, the first coordinates of the foreground pixel points in the first mask image are mapped to the camera coordinate system according to the normal line and the preset parameters of the camera to obtain the corresponding second coordinates.

[0062] For example, the normal N = {a, b, c, d}, the camera preset parameters are as follows:

[0063]

[0064] If the coordinates of all foreground pixels in the first mask image M1 are in P 1 The coordinates obtained by mapping to the camera coordinate system in:

[0065]

[0066]

[0067]

[0068]

[0069]

[0070] Next, the second coordinate is mapped to the camera plane coordinate system to obtain the corresponding third coordinate. The camera plane coordinate system is the XOY plane under the camera coordinate system.

[0071] For example, using the following formula, the second coordinate P 2 Mapped into the camera plane coordinate system to get the third coordinate

[0072]

[0073]

[0074] In the above formula (13), the transformation matrix B is used to ensure that the Z-axis coordinate of the transformed point coordinates is 0.

[0075] Next, calculate from the first coordinate P 1 To the third coordinate P 3 The first transformation submatrix is ​​used to map the first mask image to a plane coordinate system to generate a second mask image. For example, the first mask image M1 is as follows: Figure 3A As shown, the second mask image M2 is Figure 3B shown.

[0076] Next, each line segment in the first boundary line segment set is mapped to a plane coordinate system using the first transformation sub-matrix to generate a second boundary line segment set.

[0077] For example, L 3 The endpoint coordinates p of each line segment in are mapped to the camera plane coordinate system to obtain the coordinates p′, then

[0078] [p′, 1] T =A[p,1] T (15)

[0079] L 3 Each line segment in is mapped to the camera plane coordinate system to obtain the second boundary line segment set

[0080] In step 104, a minimum bounding rectangle of the object is generated in a plane coordinate system according to the second mask image and the second boundary line segment set.

[0081] In some embodiments, Figure 4 As shown, first in the second boundary segment set L 4 The line segment 4 with the largest length is selected as the reference line segment, and then the angle θ between the reference line segment 4 and the horizontal direction is calculated.

[0082] Next, the second mask image M2 is rotated by an angle θ along the first direction to generate a third mask image M3 , so that the rotated reference line segment is parallel to the horizontal direction.

[0083] For example, by Figure 4 The second mask image shown is rotated clockwise by an angle θ to obtain a third mask image M3, as shown in FIG. Figure 5 shown.

[0084] Next, the minimum bounding rectangle of the foreground area is generated in the third mask image. Figure 5 As shown in B1.

[0085] Next, the minimum bounding rectangle of the foreground area generated in the third mask image is rotated by an angle θ along a second direction opposite to the first direction to obtain the minimum bounding rectangle of the target.

[0086] For example, Figure 5 The rectangle B1 in is rotated counterclockwise by angle θ to obtain the minimum circumscribed rectangle of the target, as shown in Figure 6 As shown in rectangle B2.

[0087] In step 105, the preset template image is mapped into the minimum bounding rectangle of the target to generate a transformed image.

[0088] In some embodiments, the minimum bounding rectangle of the target is first transformed into a square. Figure 7 As shown, the minimum circumscribed rectangle B2 of the target is transformed into a square B3.

[0089] Next, a second transformation sub-matrix for mapping the template image to a square is calculated, and a transformation matrix is ​​obtained according to the first transformation sub-matrix and the second transformation sub-matrix.

[0090] Next, the template image is mapped into a pixel coordinate system according to a transformation matrix to generate a transformation image. For example, the transformation matrix is ​​the product of the inverse matrix of the first transformation submatrix and the second transformation submatrix.

[0091] If the first transformation submatrix is ​​A and the second transformation submatrix is ​​D, then the transformation matrix is ​​A -1 D.

[0092] If the template image I 1 The coordinates of a point in Then in the transformed image I 2 Coordinates in for:

[0093]

[0094] For example, the template image I 1 like Figure 8 As shown, the corresponding transformed image I 2 like Fig. 9 shown.

[0095] In step 106, the object in the scene image is replaced with the conversion image to generate a virtual image.

[0096] In some embodiments, when the pixel value of the pixel point in the i-th row and j-th column in the first mask image is greater than 0, the pixel value of the pixel point in the i-th row and j-th column in the scene image is replaced with the pixel value of the pixel point in the i-th row and j-th column in the conversion image.

[0097] That is, the scene image I is transformed into 0 The goal is to transform image I 2 , to obtain the virtual image I 3 ,like Fig.10 shown.

[0098]

[0099] Fig.11 FIG. 1 is a schematic diagram of the structure of an image processing device according to an embodiment of the present disclosure. Fig.11 As shown, the image processing device includes a first processing module 1101 , a second processing module 1102 , a third processing module 1103 , a fourth processing module 1104 , a fifth processing module 1105 and an image updating module 1106 .

[0100] The first processing module 1101 is configured to extract a first mask image of an object in a scene image and a normal line of the object in a camera coordinate system.

[0101] In some embodiments, the LiteSeg network is used to extract the scene image I 0 In addition, by extracting the intermediate layer feature f of the LiteSeg network, a global pooling operation is performed after a convolution operation to obtain a feature map with a predetermined size (for example, 1×1×1024), and finally the normal N of the target in the camera coordinate system is obtained through a fully connected layer.

[0102] The second processing module 1102 is configured to obtain a first boundary line segment set of the target in a pixel coordinate system according to the first mask image.

[0103] In some embodiments, the second processing module 1102 first performs straight line detection on the first mask image to generate a first candidate line segment set.

[0104] Next, the second processing module 1102 deletes the boundary line segments of the scene image in the first candidate line segment set to generate a second candidate line segment set.

[0105] For example, by judging the first candidate line segment set L 1 Whether the i-th line segment in satisfies the first condition or the second condition, wherein the first condition is |x1-x2|<β and |x1+x2-W|>αW, the second condition is |y1-y2|<β and |y1+y2-H|>αH, the coordinates of the two endpoints of the i-th line segment are x1, y1 and x1, y2 respectively, the size of the first mask image is H×W, α and β are preset parameters, 1≤i≤n1, and n1 is the number of candidate line segments in the first candidate line segment set. If the i-th line segment satisfies the first condition or the second condition, the i-th line segment is deleted from the first candidate line segment set.

[0106] Then, the boundary line segments of the object located in the target area in the second candidate line segment set are deleted to generate a first boundary line segment set.

[0107] For example, the second processing module 1102 determines whether the j-th line segment in the second candidate line segment set satisfies the third condition, wherein the third condition is abs(mean(reg1)-mean(reg2))<γ, mean(reg1) is the mean pixel value of the pixel points in the first area on one side of the j-th line in the minimum circumscribed convex polygon of the first mask image, mean(reg2) is the mean pixel value of the pixel points in the second area on the other side of the j-th line in the minimum circumscribed convex polygon of the first mask image, abs is an absolute value function, γ is a preset parameter, 1≤j≤n2, and n2 is the number of candidate line segments in the second candidate line segment set. If the j-th line segment satisfies the third condition, the j-th line segment is deleted from the second candidate line segment set.

[0108] The third processing module 1103 is configured to map the first mask image and each boundary line segment in the first boundary line segment set to a camera plane coordinate system according to the normal and camera preset parameters, so as to generate a second mask image and a second boundary line segment set.

[0109] In some embodiments, the third processing module 1103 first maps the first coordinates of the foreground pixel points in the first mask image to the camera coordinate system according to the normal and the preset camera parameters to obtain the corresponding second coordinates.

[0110] Next, the third processing module 1103 maps the second coordinate to the camera plane coordinate system to obtain the corresponding third coordinate. The camera plane coordinate system is the XOY plane under the camera coordinate system.

[0111] Next, the third processing module 1103 calculates a first transformation sub-matrix from the first coordinate to the third coordinate, and maps the first mask image to a plane coordinate system using the first transformation sub-matrix to generate a second mask image.

[0112] Next, the third processing module 1103 uses the first transformation sub-matrix to map each line segment in the first boundary line segment set to a plane coordinate system to generate a second boundary line segment set.

[0113] The fourth processing module 1104 is configured to generate a minimum circumscribed rectangle of the target in a plane coordinate system according to the second mask image and the second boundary line segment set.

[0114] In some embodiments, the fourth processing module 1104 first selects a line segment with the largest length from the second boundary line segment set as a reference line segment, and then calculates an angle θ between the reference line segment and the horizontal direction.

[0115] Next, the fourth processing module 1104 rotates the second mask image by an angle θ along the first direction to generate a third mask image M3 , so that the rotated reference line segment is parallel to the horizontal direction.

[0116] Next, the fourth processing module 1104 generates a minimum bounding rectangle of the foreground area in the third mask image.

[0117] Next, the fourth processing module 1104 rotates the minimum bounding rectangle of the foreground area generated in the third mask image by an angle θ along a second direction opposite to the first direction to obtain the minimum bounding rectangle of the target.

[0118] The fifth processing module 1105 is configured to map the preset template image into the minimum bounding rectangle of the target to generate a transformed image.

[0119] In some embodiments, the fifth processing module 1105 first transforms the minimum bounding rectangle of the target into a square.

[0120] Next, the fifth processing module 1105 calculates a second transformation sub-matrix for mapping the template image to a square, and obtains a transformation matrix according to the first transformation sub-matrix and the second transformation sub-matrix.

[0121] Next, the template image is mapped into a pixel coordinate system according to a transformation matrix to generate a transformation image. For example, the transformation matrix is ​​the product of the inverse matrix of the first transformation submatrix and the second transformation submatrix.

[0122] The image updating module 1106 is configured to replace the target in the scene image with the conversion image to generate a virtual image.

[0123] In some embodiments, when the pixel value of the pixel point in the i-th row and j-th column in the first mask image is greater than 0, the pixel value of the pixel point in the i-th row and j-th column in the scene image is replaced with the pixel value of the pixel point in the i-th row and j-th column in the conversion image.

[0124] Fig.12 FIG. 1 is a schematic diagram of the structure of an image processing device according to another embodiment of the present disclosure. Fig.12 As shown, the image processing apparatus includes a memory 1201 and a processor 1202 .

[0125] The memory 1201 is used to store instructions. The processor 1202 is coupled to the memory 1201. The processor 1202 is configured to execute the instructions stored in the memory to implement the following. Figure 1 The method of any one of the embodiments.

[0126] like Fig.12 As shown, the image processing device further includes a communication interface 1203 for information exchange with other devices. At the same time, the image processing device further includes a bus 1204, through which the processor 1202, the communication interface 1203, and the memory 1201 communicate with each other.

[0127] The memory 1201 may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory. The memory 1201 may also be a memory array. The memory 1201 may also be divided into blocks, and the blocks may be combined into virtual volumes according to certain rules.

[0128] In addition, the processor 1202 may be a central processing unit (CPU), or may be an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present disclosure.

[0129] The present disclosure also relates to a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, which are executed by a processor to implement the following Figure 1 The method of any one of the embodiments.

[0130] In some embodiments, the functional unit module described above can be implemented as a general-purpose processor, a programmable logic controller (PLC), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components or any appropriate combination thereof for performing the functions described in the present disclosure.

[0131] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0132] The description of the present disclosure is given for the purpose of illustration and description, and is not intended to be exhaustive or to limit the present disclosure to the disclosed form. Many modifications and variations will be apparent to those of ordinary skill in the art. The embodiments are selected and described in order to better illustrate the principles and practical applications of the present disclosure, and to enable those of ordinary skill in the art to understand the present disclosure and thereby design various embodiments with various modifications suitable for specific uses.

Claims

1. An image processing method, comprising: Extracting a first mask image of a target in a scene image and a normal of the target in a camera coordinate system; Acquire a first boundary line segment set of the target in a pixel coordinate system according to the first mask image; According to the normal line and camera preset parameters, mapping the first mask image and each boundary line segment in the first boundary line segment set to a camera plane coordinate system to generate a second mask image and a second boundary line segment set; In the camera plane coordinate system, generating a minimum circumscribed rectangle of the target according to the second mask image and the second boundary line segment set; Mapping a preset template image into a minimum bounding rectangle of the target to generate a transformed image; Replacing the target in the scene image with the transformed image to generate a virtual image; Wherein, acquiring a first boundary line segment set of the target in a pixel coordinate system according to the first mask image comprises: Performing straight line detection on the first mask image to generate a first candidate line segment set; Deleting the boundary line segments of the scene image in the first candidate line segment set to generate a second candidate line segment set; Deleting the boundary line segments of the object located in the target area from the second candidate line segment set to generate the first boundary line segment set; According to the normal line and the preset camera parameters, mapping the first mask image and each boundary line segment in the first boundary line segment set to a camera plane coordinate system includes: Mapping the first coordinates of the foreground pixel points in the first mask image to the camera coordinate system according to the normal and the preset camera parameters to obtain corresponding second coordinates; Mapping the second coordinate into the camera plane coordinate system to obtain a corresponding third coordinate; Calculating a first transformation sub-matrix from the first coordinate to the third coordinate; Mapping the first mask image to the camera plane coordinate system using the first transformation sub-matrix to generate a second mask image; Each line segment in the first boundary line segment set is mapped to the camera plane coordinate system using the first transformation sub-matrix to generate a second boundary line segment set.

2. The method according to claim 1, wherein: Deleting the boundary line segments of the scene image in the first candidate line segment set includes: Determine whether the i-th line segment in the first candidate line segment set satisfies a first condition or a second condition, wherein the first condition is |x1-x2|<β and |x1+x2-W|>αW, the second condition is |y1-y2|<β and |y1+y2-H|>αH, the coordinates of the two endpoints of the i-th line segment are x1, y1 and x1, y2 respectively, the size of the first mask image is H×W, α and β are preset parameters, 1≤i≤n1, and n1 is the number of candidate line segments in the first candidate line segment set; If the i-th line segment satisfies the first condition or the second condition, the i-th line segment is deleted from the first candidate line segment set.

3. The method according to claim 2, wherein: Deleting the boundary line segments of the object located in the target area from the second candidate line segment set includes: Determine whether the j-th line segment in the second candidate line segment set satisfies a third condition, wherein the third condition is abs(mean(reg1)-mean(reg2))<γ, mean(reg1) is the mean pixel value of the pixels in the first region on one side of the j-th line in the minimum circumscribed convex polygon of the first mask image, mean(reg2) is the mean pixel value of the pixels in the second region on the other side of the j-th line in the minimum circumscribed convex polygon of the first mask image, abs is an absolute value function, γ is a preset parameter, 1≤j≤n2, and n2 is the number of candidate line segments in the second candidate line segment set; If the j-th line segment satisfies the third condition, the j-th line segment is deleted from the second candidate line segment set.

4. The method according to claim 1, wherein: In the camera plane coordinate system, generating a minimum circumscribed rectangle of the target according to the second mask image and the second boundary line segment set includes: Selecting a line segment with the largest length from the second boundary line segment set as a reference line segment; Calculate the angle θ between the reference line segment and the horizontal direction; Rotating the second mask image by an angle θ along the first direction to generate a third mask image, so that the rotated reference line segment is parallel to the horizontal direction; Generating a minimum bounding rectangle of a foreground area in the third mask image; The minimum bounding rectangle of the foreground area generated in the third mask image is rotated by an angle θ along a second direction opposite to the first direction to obtain the minimum bounding rectangle of the object.

5. The method according to claim 1, wherein: Mapping the preset template image into the minimum bounding rectangle of the target to generate a transformed image includes: Transforming the minimum circumscribed rectangle of the target into a square; Calculating a second transformation sub-matrix for mapping the template image into the square; Obtaining a transformation matrix according to the first transformation sub-matrix and the second transformation sub-matrix; The template image is mapped into the pixel coordinate system according to the transformation matrix to generate the transformed image.

6. The method according to claim 5, wherein: The transformation matrix is ​​the product of the inverse matrix of the first transformation submatrix and the second transformation submatrix.

7. The method according to any one of claims 1 to 6, wherein: Replacing the target in the scene image with the transformed image comprises: When the pixel value of the pixel point in the i-th row and j-th column in the first mask image is greater than 0, the pixel value of the pixel point in the i-th row and j-th column in the scene image is replaced with the pixel value of the pixel point in the i-th row and j-th column in the transformed image.

8. An image processing device, comprising: A first processing module is configured to extract a first mask image of a target in a scene image and a normal of the target in a camera coordinate system; a second processing module, configured to obtain a first boundary line segment set of the target in a pixel coordinate system according to the first mask image, wherein the first mask image is subjected to straight line detection to generate a first candidate line segment set, boundary line segments of the scene image in the first candidate line segment set are deleted to generate a second candidate line segment set, and boundary line segments of objects located in the target area in the second candidate line segment set are deleted to generate the first boundary line segment set; A third processing module is configured to map the first mask image and each boundary line segment in the first boundary line segment set to a camera plane coordinate system according to the normal line and camera preset parameters to generate a second mask image and a second boundary line segment set, wherein the first coordinate of the foreground pixel point in the first mask image is mapped to the camera coordinate system according to the normal line and camera preset parameters to obtain a corresponding second coordinate, the second coordinate is mapped to the camera plane coordinate system to obtain a corresponding third coordinate, a first transformation submatrix from the first coordinate to the third coordinate is calculated, the first mask image is mapped to the camera plane coordinate system using the first transformation submatrix to generate a second mask image, and each line segment in the first boundary line segment set is mapped to the camera plane coordinate system using the first transformation submatrix to generate a second boundary line segment set; a fourth processing module, configured to generate a minimum circumscribed rectangle of the target in the camera plane coordinate system according to the second mask image and the second boundary line segment set; A fifth processing module is configured to map a preset template image into a minimum bounding rectangle of the target to generate a transformed image; The image updating module is configured to replace the target in the scene image with the transformed image to generate a virtual image.

9. An image processing device, comprising: a memory configured to store instructions; A processor is coupled to the memory, and the processor is configured to execute the method according to any one of claims 1 to 7 based on instructions stored in the memory.

10. A computer-readable storage medium, wherein: The computer-readable storage medium stores computer instructions, and when the instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images