Image processing method, device, electronic device and storage medium
By acquiring the depth information and normal vector components in the image and deformation processing on the image, the problem of poor image effects caused by the shape simulation of two-dimensional clothing in the prior art is solved, and a more realistic three-dimensional shape replacement effect is achieved.
Patent Information
- Application Number
- CN202010608415.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-06-29
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2040-06-29
AI Technical Summary
The prior art when changing the body in the image virtually, only simulating the shape of the clothing on the two-dimensional plane, resulting in poor image area replacement effect and lacking the authenticity of the three-dimensional scene.
By acquiring visible light images and depth images, normal vector components of the target replacement area are determined, and these components are used to deform the replacement pattern, taking into account the shapes in the three-dimensional scene, thereby generating a target pattern closer to the real shape and covering it to the image middle.
The image effect after image area replacement is improved, making the replaced image more realistic and close to the three-dimensional scene.
Smart Images

Figure CN113936026B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, device, electronic device and storage medium. Background Art
[0002] With the development of computer vision technology, especially the emergence of deep neural networks, it has become possible to automatically replace specified targets in images based on computers.
[0003] Replace the specified target in the image. A typical scenario is to virtually change the clothes of the human body in the image. In the prior art, a deep neural network is used to identify the position and shape of the human body in the image, and then generate clothing in the shape of the human body and cover the corresponding position of the human body, thereby completing the human body change.
[0004] However, the above method only simulates the shape of the clothing in the two-dimensional plane of the image, resulting in poor image effect after the image area is replaced. Summary of the invention
[0005] The purpose of the embodiments of the present application is to provide an image processing method, device, electronic device and storage medium to improve the image effect after image area replacement. The specific technical solution is as follows:
[0006] In a first aspect, an embodiment of the present application provides an image processing method, the method comprising:
[0007] Acquire a visible light image to be processed and a depth image corresponding to the visible light image, wherein the visible light image includes a target to be processed;
[0008] Determining the shape of the replacement pattern and the target replacement area according to the position area of the target to be processed in the visible light image;
[0009] Calculate the normal vector component of each point in the target replacement area according to the target replacement area and the depth image, wherein each point corresponds to one or more pixel points;
[0010] According to the normal vector components of each point in the target replacement area, the shape of the replacement pattern is adjusted to obtain a target pattern;
[0011] The target pattern is overlaid into the target replacement area of the visible light image.
[0012] In a possible implementation, determining the shape of the replacement pattern and the target replacement area according to the position area of the target to be processed in the visible light image includes:
[0013] According to a tripartite map of the visible light image and the visible light image, determining a location area of the target to be processed in the visible light image as a target area to be processed, wherein the tripartite map includes foreground pixels, background pixels, and uncertain area pixels;
[0014] According to the key points in the target area to be processed, the shape of the target replacement area and the replacement pattern are determined.
[0015] In a possible implementation, determining, based on the tripartite graph of the visible light image and the visible light image, a location region of the target to be processed in the visible light image as the target region to be processed includes:
[0016] Segmenting the visible light image into foreground pixels and background pixels to obtain a segmented image, wherein the target to be processed corresponds to the foreground pixels;
[0017] Performing edge detection on the segmented image to determine the edge image area of the foreground pixel;
[0018] Performing a dilation operation on the edge image region in the segmented image to obtain pixels in an uncertain region;
[0019] The tripartite map including the foreground pixels, the background pixels and the uncertain area pixels and the visible light image are input into a pre-trained deep neural network for analysis to obtain the position area of the target to be processed in the visible light image as the target area to be processed.
[0020] In a possible implementation, calculating the normal vector component of each point in the target replacement area according to the target replacement area and the depth image includes:
[0021] Determining depth information of each point in the target replacement area according to the depth image;
[0022] For each point in the target replacement area, determine the area range of the current point with the current point as the center and a preset length as the radius;
[0023] According to the depth information of each point within the domain of the current point, each point within the domain of the current point is fitted into a local plane;
[0024] The normal vector component of the local plane of the current point is calculated as the normal vector component of the current point, and then the normal vector component of each point in the target replacement area is obtained.
[0025] In a possible implementation manner, the normal vector components of each point in the target replacement area include horizontal and vertical coordinate components in the two-dimensional coordinate system direction of the visible light image;
[0026] The step of adjusting the shape of the replacement pattern according to the normal vector components of each point in the target replacement area to obtain the target pattern includes:
[0027] Determining the deformation direction and deformation intensity of each corresponding point in the replacement pattern according to the direction and length of the horizontal and vertical coordinate components of each point in the target replacement area;
[0028] According to the deformation direction and deformation strength of each corresponding point in the replacement pattern, each point in the replacement pattern is adjusted to obtain a target pattern.
[0029] In a second aspect, an embodiment of the present application provides an image processing device, the device comprising:
[0030] An image acquisition module, used to acquire a visible light image to be processed and a depth image corresponding to the visible light image, wherein the visible light image includes a target to be processed;
[0031] An area determination module, used to determine the shape of the replacement pattern and the target replacement area according to the position area of the target to be processed in the visible light image;
[0032] A normal vector component calculation module, used to calculate the normal vector component of each point in the target replacement area according to the target replacement area and the depth image, wherein each of the points corresponds to one or more pixel points;
[0033] An image deformation processing module, used for adjusting the shape of the replacement pattern according to the normal vector components of each point in the target replacement area to obtain a target pattern;
[0034] The target area replacement module is used to cover the target pattern into the target replacement area of the visible light image.
[0035] In a possible implementation manner, the region determination module includes:
[0036] A first region determination submodule is used to determine, according to a tripartite map of the visible light image and the visible light image, a location region of the target to be processed in the visible light image as a target region to be processed, wherein the tripartite map includes foreground pixels, background pixels, and uncertain region pixels;
[0037] The second region determination submodule is used to determine the target replacement region and the shape of the replacement pattern according to the key points in the target region to be processed.
[0038] In a possible implementation, the first region determination submodule is specifically used to: segment the visible light image into foreground pixels and background pixels to obtain a segmented image, wherein the target to be processed corresponds to the foreground pixels; perform edge detection on the segmented image to determine the edge image region of the foreground pixels; perform a dilation operation on the edge image region in the segmented image to obtain uncertain region pixels; and input a tripartite map including the foreground pixels, the background pixels, and the uncertain region pixels and the visible light image into a pre-trained deep neural network for analysis to obtain a position region of the target to be processed in the visible light image as the target region to be processed.
[0039] In a possible implementation, the normal vector component calculation module is specifically used to: determine the depth information of each point in the target replacement area based on the depth image; for each point in the target replacement area, determine the domain range of the current point with the current point as the center and a preset length as the radius; fit each point within the domain range of the current point into a local plane based on the depth information of each point within the domain range of the current point; calculate the normal vector component of the local plane of the current point as the normal vector component of the current point, and thereby obtain the normal vector component of each point in the target replacement area.
[0040] In a possible implementation, the normal vector components of each point in the target replacement area include horizontal and vertical coordinate components in the two-dimensional coordinate system of the visible light image; the image deformation processing module is specifically used to: determine the deformation direction and deformation strength of each corresponding point in the replacement pattern according to the direction and length of the horizontal and vertical coordinate components of each point in the target replacement area; adjust each point in the replacement pattern according to the deformation direction and deformation strength of each corresponding point in the replacement pattern to obtain a target pattern.
[0041] In a third aspect, an embodiment of the present application provides an electronic device, including a processor and a memory;
[0042] The memory is used to store computer programs;
[0043] The processor is used to implement any of the above-mentioned image processing methods when executing the program stored in the memory.
[0044] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the computer program implements any of the above-mentioned image processing methods.
[0045] The image processing method, device, electronic device and storage medium provided in the embodiments of the present application obtain a visible light image to be processed and a depth image corresponding to the visible light image, wherein the visible light image includes a target to be processed; determine the shape of the replacement pattern and the target replacement area according to the position area of the target to be processed in the visible light image; calculate the normal vector components of each point in the target replacement area according to the target replacement area and the depth image, wherein each point corresponds to one or more pixel points; adjust the shape of the replacement pattern according to the normal vector components of each point in the target replacement area to obtain the target pattern; and cover the target pattern in the target replacement area of the visible light image. Using the depth information in the depth image, calculate the normal vector components of each point in the target replacement area that needs to be replaced, and use the normal vector components of each point to deform the surface of the replacement pattern, taking into account the shape of the target in the three-dimensional scene, the obtained target pattern is closer to the real shape, and the image effect after replacement is more realistic. Of course, it is not necessary to achieve all of the above advantages at the same time to implement any product or method of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0047] Figure 1 A schematic diagram of an image processing method according to an embodiment of the present application;
[0048] Figure 2 A schematic diagram of a method for determining a target replacement area according to an embodiment of the present application;
[0049] Figure 3 A schematic diagram of a method for determining a target area to be processed according to an embodiment of the present application;
[0050] Figure 4 A schematic diagram of a method for determining a normal vector component according to an embodiment of the present application;
[0051] Figure 5 A schematic diagram of a method for determining a target pattern according to an embodiment of the present application;
[0052] Figure 6 A schematic diagram of an image processing device according to an embodiment of the present application;
[0053] Figure 7 A schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION
[0054] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0055] First, the professional terms in the embodiments of the present application are explained:
[0056] Depth image: In 3D (three-dimensional) computer graphics, a depth image contains information about the distance to the surface of scene objects. A depth image is similar to a grayscale image, except that each pixel value is the actual distance from the sensor to the object. Usually RGB (Red-Green-Blue) images and depth images are registered, so there is a one-to-one correspondence between pixels.
[0057] Normal vector: It is a concept in spatial analytic geometry. The vector represented by a straight line perpendicular to a plane is the normal vector of the plane.
[0058] In order to improve the image effect after image region replacement, an embodiment of the present application provides an image processing method, the method comprising:
[0059] Acquire a visible light image to be processed and a depth image corresponding to the visible light image, wherein the visible light image includes the target to be processed;
[0060] Determine the position area of the target to be processed in the visible light image to obtain the target area to be processed;
[0061] According to the above-mentioned target area to be processed, determining the shape of the replacement pattern and the target replacement area;
[0062] Calculate the normal vector components of each point in the target replacement area according to the target replacement area and the depth image, wherein each of the points corresponds to one or more pixel points;
[0063] According to the normal vector components of each point in the target replacement area, the surface of the replacement pattern is deformed to obtain a target pattern;
[0064] The target pattern is overlaid into the target replacement area of the visible light image.
[0065] In an embodiment of the present application, the depth information in the depth image is used to obtain the normal vector components of each point in the target replacement area to be replaced, and the normal vector components of each point are used to deform the surface of the replacement pattern. The shape of the target in the three-dimensional scene is taken into consideration, and the obtained target pattern is closer to the real shape, and the image effect after replacement is more realistic.
[0066] The following is a detailed description, see Figure 1 , Figure 1 This is a first schematic diagram of an image processing method according to an embodiment of the present application, the method comprising:
[0067] S11, obtaining a visible light image to be processed and a depth image corresponding to the visible light image, wherein the visible light image includes a target to be processed.
[0068] The image processing method of the embodiment of the present application can be implemented by an electronic device with image processing capabilities. Specifically, the electronic device can be a server, a personal computer, a smart phone, or smart glasses.
[0069] The visible light image can be an image in any color mode, such as an RGB image or a YUV image. There is a one-to-one correspondence between each pixel in the visible light image and the pixel in the depth image. The target to be processed is the target that needs to be replaced partially or completely. For example, for a dress-changing scene, the target to be processed is a human body; for a face-changing scene, the target to be processed is a human face, etc.
[0070] S12, determining the shape of the replacement pattern and the target replacement area according to the position area of the target to be processed in the visible light image.
[0071] Computer vision technology can be used to determine the location of the target to be processed in the visible light image, so as to further obtain the shape of the replacement pattern and the target replacement area. The computer vision technology here includes but is not limited to computer vision technology based on deep learning network, computer vision technology based on feature transformation comparison, and computer vision technology based on foreground and background recognition.
[0072] The target to be processed corresponds to a pixel area in the visible light image and has a certain shape. The area to be replaced, i.e., the target replacement area, is determined according to the position area of the target to be processed in the visible light image. The shape of the target replacement area is the shape of the replacement pattern. The replacement pattern is the pattern that the target replacement area needs to be replaced with. For example, for a dress-changing scene, the target replacement area can be a clothing area, and the replacement pattern is the clothing pattern that needs to be replaced. For example, for a face-changing scene, the target replacement area can be a human face area, and the replacement pattern is the human face pattern that needs to be replaced.
[0073] S13, calculating normal vector components of each point in the target replacement area according to the target replacement area and the depth image, wherein each of the points corresponds to one or more pixel points.
[0074] The point in the target replacement area is the basic unit for calculating the normal vector component. A point can correspond to one or more pixels. When the size of the target replacement area remains unchanged, the more pixels a point corresponds to, the fewer normal vector components need to be calculated, and the fewer computing resources are consumed, but the quality of the target pattern will be lower; the fewer pixels a point corresponds to (at least one pixel), the more normal vector components need to be calculated, and the more computing resources are consumed, but the quality of the target pattern will be higher. Here you can customize the number of pixels corresponding to the point according to actual conditions.
[0075] S14, adjusting the shape of the replacement pattern according to the normal vector components of each point in the target replacement area to obtain a target pattern.
[0076] Each point in the replacement pattern can be mapped to a three-dimensional image in three-dimensional space according to the normal vector component of each point, and then the projection of the replacement pattern on the visible light image is taken to obtain the target pattern. Alternatively, the normal vector component of each point on the visible light image can be taken, and the replacement pattern can be deformed using the normal vector component of each point on the visible light image to obtain the target pattern. For the costume change scene, the target pattern is the deformed clothing.
[0077] S15, covering the target pattern into the target replacement area of the visible light image.
[0078] The target pattern is overlaid into the target replacement area of the visible light image to complete the replacement.
[0079] In an embodiment of the present application, the depth information in the depth image is used to obtain the normal vector components of each point in the target replacement area to be replaced, and the normal vector components of each point are used to deform the surface of the replacement pattern. The shape of the target in the three-dimensional scene is taken into consideration, and the obtained target pattern is closer to the real shape, and the image effect after replacement is more realistic.
[0080] In one possible implementation, see Figure 2 The above-mentioned determining the shape of the replacement pattern and the target replacement area according to the position area of the above-mentioned target to be processed in the above-mentioned visible light image includes:
[0081] S121, determining a location area of the target to be processed in the visible light image as a target area to be processed according to the triplicate map of the visible light image and the visible light image, wherein the triplicate map includes foreground pixels, background pixels and uncertain area pixels.
[0082] The tripartite map of a visible light image refers to dividing the pixels in the visible light image into three parts, namely foreground pixels, background pixels, and uncertain area pixels. Among them, the foreground pixels correspond to the pixels of the target to be processed, the background pixels correspond to the pixels other than the target to be processed, and the uncertain area pixels correspond to the pixels that are uncertain whether they are the target to be processed.
[0083] S122, determining the shape of the target replacement area and the replacement pattern according to the key points in the target area to be processed.
[0084] The target to be processed in the target area to be processed is identified by key points to obtain a plurality of key points. According to each of the key points, the target replacement area and the shape of the replacement pattern are determined.
[0085] The target replacement area can be a part of the target area to be processed. For the case where the target replacement area is a part of the target area to be processed, the target replacement area can be determined by key point recognition. For example, for a dressing scene, the target area to be processed is the human body area, the target replacement area is the clothing area in the human body area, and the replacement pattern is the clothing pattern that needs to be replaced. According to the key points of the human body area, the clothing area, i.e., the target replacement area, is determined, and the shape of the target replacement area is the shape of the replacement pattern.
[0086] In the embodiment of the present application, in the case where the target replacement area is a part of the target area to be processed, the shape of the target replacement area and the replacement pattern can be quickly and accurately determined by a key point recognition method.
[0087] In one possible implementation, see Figure 3 , the step of determining the position area of the target to be processed in the visible light image according to the trisection image of the visible light image and the visible light image as the target area to be processed includes:
[0088] S1211, segmenting the visible light image into foreground pixels and background pixels to obtain a segmented image, wherein the target to be processed corresponds to the foreground pixel.
[0089] Foreground pixels refer to pixels corresponding to the target to be processed. For example, for a dressing scene, foreground pixels are pixels corresponding to the human body. Relevant image segmentation algorithms, such as graph cut algorithm, watershed algorithm, and image segmentation algorithm based on deep learning, can be used to segment the foreground pixels and background pixels of the visible light image. Optionally, the foreground pixels and background pixels can be determined according to the foreground pixel and background pixel selection instructions input by the user, and then the segmented image can be obtained using the graph cut algorithm. The pixel value of the foreground pixel can be set to 255, and the pixel value of the background pixel can be set to 0.
[0090] S1212, performing edge detection on the segmented image to determine the edge image region of the foreground pixel.
[0091] The foreground pixels and background pixels in the segmented image may not be completely accurate. The segmented image is subjected to edge detection through a related edge detection algorithm to obtain the distinguishing area between the foreground pixels and the background pixels, namely the edge image area.
[0092] S1213, performing a dilation operation on the edge image region in the segmented image to obtain pixels in the uncertain region.
[0093] The dilation operation refers to increasing the range of the edge image area. In general, the edge area is a wireframe, and the width of the specified number of pixels can be diffused to the inside and outside of the wireframe at the same time, so as to perform a dilation operation on the edge image area. The pixels corresponding to the edge image area after the dilation operation are the pixels in the uncertain area. The pixel value of the pixels in the uncertain area can be set to 128.
[0094] S1214, inputting the tripartite map including the foreground pixels, the background pixels and the uncertain area pixels and the visible light image into a pre-trained deep neural network for analysis, and obtaining the position area of the target to be processed in the visible light image as the target area to be processed.
[0095] The three-part image input to the deep neural network includes foreground pixels, background pixels and uncertain area pixels. The deep neural network analyzes the input three-part image and visible light image to obtain the accurate location area of the target to be processed in the visible light image, that is, the target area to be processed.
[0096] The above-mentioned deep neural network can be obtained by training based on the sample tripartite map and the sample visible light image marked with the true value of the target area to be processed. Optionally, the training process of the pre-trained deep neural network may include: obtaining an initial deep neural network, where the deep neural network may include a codec network and a correction network, wherein the codec network is used to encode and decode the input image, and the correction network is used to correct the result. The sample tripartite map and the sample visible light image including three parts of foreground pixels, background pixels and uncertain area pixels are input into the deep neural network for training, wherein the sample visible light image is marked with the true value of the target area to be processed, and the sample tripartite map is obtained based on the sample visible light image. The loss of the deep neural network is calculated based on the prediction result output by the deep neural network and the true value of the target area to be processed, and the parameters of the deep neural network are adjusted based on the loss to continue to select the sample tripartite map and the sample visible light image for training until the loss of the deep neural network converges to obtain the pre-trained deep neural network.
[0097] In an embodiment of the present application, a tripartite map including foreground pixels, background pixels and uncertain area pixels and a visible light image are analyzed through a deep neural network to obtain a target area to be processed. Compared with only analyzing the visible light image, the obtained target area to be processed is more accurate.
[0098] In one possible implementation, see Figure 4 , the above-mentioned calculation of the normal vector components of each point in the above-mentioned target replacement area according to the above-mentioned target replacement area and the above-mentioned depth image includes:
[0099] S141, determining depth information of each point in the target replacement area according to the depth image.
[0100] The pixel points in the depth image and the pixel points in the visible light image can be in a one-to-one correspondence. Based on the depth information of the corresponding pixel points in the depth image, the depth information of each point in the target replacement area can be obtained. Here, the point is used as the basic unit for calculating the normal vector component, and one point corresponds to one or more pixels. When a point corresponds to a pixel, that is, a point is a pixel, it is obvious that the depth information of the point can be obtained. When a point corresponds to multiple pixels, the average of the depths of the pixels corresponding to the point can be taken as the depth information of the point.
[0101] S142, for each point in the target replacement area, determine the area range of the point with the current point as the center and a preset length as the radius.
[0102] S143, fitting each point within the range of the current point into a local plane according to the depth information of each point within the range of the current point.
[0103] S144, calculating the normal vector component of the local plane of the current point as the normal vector component of the current point, and then obtaining the normal vector component of each point in the above-mentioned target replacement area.
[0104] Each point in the target replacement area has two-dimensional coordinate information on the visible light image. Taking a point as an example, the area of the point is determined with the point as the center and the preset length as the radius on the visible light image. The least squares method is used to fit each point in the area of the point into a local plane in three-dimensional space. The normal vector components of the local plane of the point are calculated, including the horizontal and vertical coordinate components (components in the two-dimensional plane composed of the X-axis and the Y-axis) and the Z component, wherein the Z component is the component in the depth direction, and the horizontal and vertical coordinate components are the components on the two-dimensional coordinates of the visible light image. The above operation is performed for each point, so that the normal vector components of each point in the target replacement area can be obtained. The preset length here can be set according to the actual situation, for example, to three pixels or five pixels.
[0105] In the embodiment of the present application, a local plane is fitted to each point within the domain of the point in three-dimensional space, and the normal vector component of the local plane of the point is calculated as the normal vector component of the point. The normal vector component of a point is related to the depth of the points around it, so that the image after deformation processing is softer and the image effect is better.
[0106] In a possible implementation, the normal vector components of each point in the target replacement area include horizontal and vertical coordinate components in the two-dimensional coordinate system of the visible light image; see Figure 5 , the shape of the replacement pattern is adjusted according to the normal vector components of each point in the target replacement area to obtain the target pattern, including:
[0107] S151, determining the deformation direction and deformation strength of each corresponding point in the replacement pattern according to the direction and length of the horizontal and vertical coordinate components of each point in the target replacement area.
[0108] S152, adjusting each point in the replacement pattern according to the deformation direction and deformation strength of each corresponding point in the replacement pattern to obtain a target pattern.
[0109] The horizontal and vertical coordinate components are the components in the two-dimensional coordinate system of the visible light image. The deformation direction of a point is the same as the direction of the horizontal and vertical coordinate components of the point, and the deformation intensity of a point is positively correlated with the length of the component of the point.
[0110] In the embodiment of the present application, the horizontal and vertical coordinate components of each point in the target replacement area are used to perform deformation processing on each point in the replacement pattern, and the shape of the target in the three-dimensional scene is taken into consideration. The obtained target pattern is closer to the real shape, and the replaced image effect is more realistic.
[0111] The present application also provides an image processing device. Figure 6 , the device comprises:
[0112] An image acquisition module 601 is used to acquire a visible light image to be processed and a depth image corresponding to the visible light image, wherein the visible light image includes a target to be processed;
[0113] The region determination module 602 is used to determine the shape of the replacement pattern and the target replacement region according to the position region of the target to be processed in the visible light image;
[0114] A normal vector component calculation module 603 is used to calculate the normal vector component of each point in the target replacement area according to the target replacement area and the depth image, wherein each of the points corresponds to one or more pixel points;
[0115] An image deformation processing module 604 is used to adjust the shape of the replacement pattern according to the normal vector components of each point in the target replacement area to obtain a target pattern;
[0116] The target region replacement module 605 is used to cover the target pattern into the target replacement region of the visible light image.
[0117] In a possible implementation manner, the region determination module 602 includes:
[0118] A first region determination submodule is used to determine, according to a tripartite map of the visible light image and the visible light image, a location region of the target to be processed in the visible light image as a target region to be processed, wherein the tripartite map includes foreground pixels, background pixels, and uncertain region pixels;
[0119] The second region determination submodule is used to determine the target replacement region and the shape of the replacement pattern according to the key points in the target region to be processed.
[0120] In a possible implementation, the first region determination submodule is specifically used to: segment the visible light image into foreground pixels and background pixels to obtain a segmented image, wherein the target to be processed corresponds to the foreground pixels; perform edge detection on the segmented image to determine the edge image region of the foreground pixels; perform a dilation operation on the edge image region in the segmented image to obtain uncertain region pixels; input a tripartite map including the foreground pixels, the background pixels, and the uncertain region pixels and the visible light image into a pre-trained deep neural network for analysis to obtain a position region of the target to be processed in the visible light image as the target region to be processed.
[0121] In a possible implementation, the normal vector component calculation module 603 is specifically used to: determine the depth information of each point in the target replacement area based on the depth image; for each point in the target replacement area, determine the area of the point with the current point as the center and a preset length as the radius; fit each point in the area of the current point into a local plane based on the depth information of each point in the area of the current point; calculate the normal vector component of the local plane of the current point as the normal vector component of the current point, and then obtain the normal vector component of each point in the target replacement area.
[0122] In a possible implementation, the normal vector components of each point in the target replacement area include the horizontal and vertical coordinate components in the two-dimensional coordinate system of the visible light image; the image deformation processing module 604 is specifically used to: determine the deformation direction and deformation strength of each corresponding point in the replacement pattern according to the direction and length of the horizontal and vertical coordinate components of each point in the target replacement area; adjust each point in the replacement pattern according to the deformation direction and deformation strength of each corresponding point in the replacement pattern to obtain the target pattern.
[0123] The embodiment of the present application also provides an electronic device, including: a processor and a memory;
[0124] The memory is used to store computer programs;
[0125] When the processor is used to execute the computer program stored in the memory, the following steps are implemented:
[0126] Acquire a visible light image to be processed and a depth image corresponding to the visible light image, wherein the visible light image includes the target to be processed;
[0127] Determining the shape of the replacement pattern and the target replacement area according to the position area of the target to be processed in the visible light image;
[0128] Calculate the normal vector components of each point in the target replacement area according to the target replacement area and the depth image, wherein each of the points corresponds to one or more pixel points;
[0129] According to the normal vector components of each point in the target replacement area, the shape of the replacement pattern is adjusted to obtain a target pattern;
[0130] The target pattern is overlaid into the target replacement area of the visible light image.
[0131] Optional, see Figure 7 The electronic device of the embodiment of the present application further includes a communication interface 702 and a communication bus 704 , wherein the processor 701 , the communication interface 702 , and the memory 703 communicate with each other via the communication bus 704 .
[0132] Optionally, when the processor is used to execute the computer program stored in the memory, it can also implement any of the above-mentioned image processing methods.
[0133] The communication bus mentioned in the above electronic device may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.
[0134] The communication interface is used for communication between the above electronic device and other devices.
[0135] The memory may include RAM (Random Access Memory) or NVM (Non-Volatile Memory), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.
[0136] The above-mentioned processor can be a general-purpose processor, including a CPU (Central Processing Unit), an NP (Network Processor), etc.; it can also be a DSP (Digital Signal Processing), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
[0137] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, any of the above-mentioned image processing methods is implemented.
[0138] It should be noted that, in this article, the technical features in each optional solution can be combined to form a solution as long as there is no contradiction, and these solutions are all within the scope disclosed in this application. Relational terms such as first and second, etc. are only used to distinguish an entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the term "include", "comprise" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements not only include those elements, but also include other elements that are not clearly listed, or also include elements inherent to such process, method, article or equipment. In the absence of more restrictions, the elements limited by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or equipment including the elements.
[0139] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the embodiments of the device, electronic device, and storage medium, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.
[0140] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.
Claims
1. An image processing method, It is characterized in that The method comprises: Acquire a visible light image to be processed and a depth image corresponding to the visible light image, wherein the visible light image includes a target to be processed; Determining the shape of the replacement pattern and the target replacement area according to the position area of the target to be processed in the visible light image; Calculate the normal vector component of each point in the target replacement area according to the target replacement area and the depth image, wherein each point corresponds to one or more pixel points; According to the normal vector components of each point in the target replacement area, the shape of the replacement pattern is adjusted to obtain a target pattern; The target pattern is overlaid into the target replacement area of the visible light image.
2. The method according to claim 1, It is characterized in that The step of determining the shape of the replacement pattern and the target replacement area according to the position area of the target to be processed in the visible light image includes: According to a tripartite map of the visible light image and the visible light image, determining a location area of the target to be processed in the visible light image as a target area to be processed, wherein the tripartite map includes foreground pixels, background pixels, and uncertain area pixels; According to the key points in the target area to be processed, the shape of the target replacement area and the replacement pattern are determined.
3. The method according to claim 2, It is characterized in that The step of determining, based on the trisection image of the visible light image and the visible light image, a location region of the target to be processed in the visible light image as the target region to be processed includes: Segmenting the visible light image into foreground pixels and background pixels to obtain a segmented image, wherein the target to be processed corresponds to the foreground pixels; Performing edge detection on the segmented image to determine the edge image area of the foreground pixel; Performing a dilation operation on the edge image region in the segmented image to obtain pixels in an uncertain region; The tripartite map including the foreground pixels, the background pixels and the uncertain area pixels and the visible light image are input into a pre-trained deep neural network for analysis to obtain the position area of the target to be processed in the visible light image as the target area to be processed.
4. The method according to claim 1, It is characterized in that The calculating, according to the target replacement area and the depth image, a normal vector component of each point in the target replacement area includes: Determining depth information of each point in the target replacement area according to the depth image; For each point in the target replacement area, determine the area range of the current point with the current point as the center and a preset length as the radius; According to the depth information of each point within the domain of the current point, each point within the domain of the current point is fitted into a local plane; The normal vector component of the local plane of the current point is calculated as the normal vector component of the current point, and then the normal vector component of each point in the target replacement area is obtained.
5. The method according to claim 1, It is characterized in that The normal vector components of each point in the target replacement area include horizontal and vertical coordinate components in the two-dimensional coordinate system of the visible light image; The step of adjusting the shape of the replacement pattern according to the normal vector components of each point in the target replacement area to obtain the target pattern includes: Determining the deformation direction and deformation intensity of each corresponding point in the replacement pattern according to the direction and length of the horizontal and vertical coordinate components of each point in the target replacement area; According to the deformation direction and deformation strength of each corresponding point in the replacement pattern, each point in the replacement pattern is adjusted to obtain a target pattern.
6. An image processing device, It is characterized in that The device comprises: An image acquisition module, used to acquire a visible light image to be processed and a depth image corresponding to the visible light image, wherein the visible light image includes a target to be processed; An area determination module, used to determine the shape of the replacement pattern and the target replacement area according to the position area of the target to be processed in the visible light image; A normal vector component calculation module, used to calculate the normal vector component of each point in the target replacement area according to the target replacement area and the depth image, wherein each of the points corresponds to one or more pixel points; An image deformation processing module, used for adjusting the shape of the replacement pattern according to the normal vector components of each point in the target replacement area to obtain a target pattern; The target area replacement module is used to cover the target pattern into the target replacement area of the visible light image.
7. The device according to claim 6, It is characterized in that The area determination module comprises: A first region determination submodule is used to determine, according to a tripartite map of the visible light image and the visible light image, a location region of the target to be processed in the visible light image as a target region to be processed, wherein the tripartite map includes foreground pixels, background pixels, and uncertain region pixels; The second region determination submodule is used to determine the target replacement region and the shape of the replacement pattern according to the key points in the target region to be processed.
8. The device according to claim 7, It is characterized in that The first region determination submodule is specifically used to: segment the visible light image into foreground pixels and background pixels to obtain a segmented image, wherein the target to be processed corresponds to the foreground pixels; perform edge detection on the segmented image to determine the edge image region of the foreground pixels; perform a dilation operation on the edge image region in the segmented image to obtain uncertain region pixels; input a tripartite map including the foreground pixels, the background pixels, and the uncertain region pixels and the visible light image into a pre-trained deep neural network for analysis to obtain a position region of the target to be processed in the visible light image as the target region to be processed.
9. The device according to claim 6, It is characterized in that The normal vector component calculation module is specifically used to: determine the depth information of each point in the target replacement area according to the depth image; for each point in the target replacement area, determine the area range of the current point with the current point as the center and a preset length as the radius; According to the depth information of each point within the range of the current point, each point within the range of the current point is fitted into a local plane; the normal vector component of the local plane of the current point is calculated as the normal vector component of the current point, and then the normal vector component of each point in the target replacement area is obtained.
10. The device according to claim 6, It is characterized in that The normal vector components of each point in the target replacement area include horizontal and vertical coordinate components in the two-dimensional coordinate system of the visible light image; the image deformation processing module is specifically used to: determine the deformation direction and deformation strength of each corresponding point in the replacement pattern according to the direction and length of the horizontal and vertical coordinate components of each point in the target replacement area; According to the deformation direction and deformation strength of each corresponding point in the replacement pattern, each point in the replacement pattern is adjusted to obtain a target pattern.
11. An electronic device, It is characterized in that including a processor and a memory; The memory is used to store computer programs; The processor is used to implement the image processing method described in any one of claims 1-5 when executing the program stored in the memory.
12. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the image processing method according to any one of claims 1 to 5 is implemented.
