Image transformation method, device, electronic device and storage medium

Abstract

The present disclosure relates to an image transformation method, apparatus, electronic device, and storage medium. The image transformation method includes: obtaining an original image and an optical flow mapping relationship between the original image and a target image; determining depth information of the original image; and determining a target image based on the original image, the optical flow mapping relationship, and the depth information of the original image. This disclosure can avoid pixel mapping defects in the target image caused by uneven distribution of pixels mapped from the original image.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to an image transformation method, device, electronic device, and storage medium. Background Art

[0002] The transformation of an image can be obtained through image pixel mapping. When performing image pixel mapping, a mapping relationship between the original pixel points in the original image and the target pixel points in the target image can be established, and image pixel mapping can be performed through the mapping relationship. General mapping relationships include two different mapping relationships. One mapping relationship is a forward mapping relationship, that is, by calculating the coordinate position of any original pixel point in the original image after mapping to the target image, the mapping relationship between the original pixel point and the target pixel point is obtained. The other mapping relationship is a backward mapping relationship, that is, by calculating the position of any target pixel point in the transformed target image in the original image, the mapping relationship between the original pixel point and the target pixel point is obtained.

[0003] If the mapping relationship in the related art is used, the original pixel points will be unevenly mapped, resulting in inaccurate determination of the target pixel points. Summary of the Invention

[0004] To overcome the problems existing in the related art, the present disclosure provides an image transformation method, device, electronic device and storage medium.

[0005] According to a first aspect of an embodiment of the present disclosure, there is provided an image transformation method, comprising:

[0006] Acquire an original image and an optical flow mapping relationship between the original image and a target image; determine depth information of the original image; and determine a target image based on the original image, the optical flow mapping relationship, and the depth information of the original image.

[0007] In one embodiment, the optical flow mapping relationship is a forward optical flow mapping relationship;

[0008] The determining of the target image based on the original image, the optical flow mapping relationship, and the depth information of the original image includes:

[0009] Based on the forward optical flow mapping relationship, each original pixel point in the original image is mapped to the target image to obtain a mapping pixel point; among the mapping pixel points, the mapping pixel points located within the set range of the specified target pixel point position are determined; based on the depth information, among the mapping pixel points within the set range, the unobstructed mapping pixel points are screened out, and the target image is determined based on the unobstructed mapping pixel points.

[0010] In one embodiment, the selecting unobstructed mapped pixel points from the mapped pixel points within a set range based on the depth information includes:

[0011] Based on the depth information, an occlusion relationship between mapping pixel points within a set range is determined; based on the occlusion relationship, a weight of each mapping pixel point is determined; wherein, a first mapping pixel point occludes a second mapping pixel point, and the weight of the first mapping pixel point is greater than the weight of the second mapping pixel point; and mapping pixel points whose weights of the mapping pixel points are greater than a weight threshold are determined as unoccluded mapping pixel points.

[0012] In one embodiment, determining the target image based on the unobstructed mapped pixel points includes:

[0013] Based on the distance between the unobstructed mapping pixel points, determine the weight of the distance from the position of each mapping pixel point in the unobstructed mapping pixel points to the designated pixel position of the target image; wherein, the distance from the first mapping pixel point to the designated pixel position of the target image is less than the distance from the second mapping pixel point to the designated pixel position of the target image, and the weight of the first mapping pixel point is greater than the weight of the second mapping pixel point; based on the weight from the position of each mapping pixel point to the designated pixel position, perform an interpolation operation on the unobstructed mapping pixel points to obtain the target pixel point corresponding to the designated pixel position; repeat the above process until the target pixel points of all pixel positions in the target image are obtained to obtain the target image.

[0014] In one embodiment, the optical flow mapping relationship is a backward optical flow mapping relationship;

[0015] The determining of the target image based on the original image, the optical flow mapping relationship, and the depth information of the original image includes:

[0016] Based on the backward optical flow mapping relationship, the optical flow of the specified target pixel point position in the target image is mapped to the original image; among the original pixel points in the original image, the original pixel points located within the set range of the position mapped by the specified target pixel point position are determined; based on the original pixel points within the set range, the target image is determined.

[0017] In one embodiment, determining the target image based on the original pixel points within the set range includes:

[0018] Based on the distances between the original pixel points, the weight of the distance from the position of each original pixel point in the original pixel points to the mapped position of the specified target pixel point is determined; wherein, the distance from the first original pixel point to the mapped position of the target pixel point is smaller than the distance from the second original pixel point to the mapped position of the target pixel point, and the weight of the first original pixel point is larger than the weight of the second original pixel point; based on the weight from the position of each original pixel point to the specified pixel position, the original pixel points are interpolated to obtain the target pixel point corresponding to the specified pixel position; the above process is repeated until the target pixel points of all pixel positions in the target image are obtained to obtain the target image.

[0019] In one embodiment, the setting range is determined as follows:

[0020] A set range is determined based on a first set length and a designated target pixel position; if the original pixel exists within the set range, the step of determining the set range is terminated; if the original pixel does not exist within the set range, the set range is re-determined based on a second set length and a designated target pixel position, and it is determined whether the original pixel exists within the re-determined set range; the step of determining the set range is repeated based on different set lengths until the original pixel exists within the re-determined set range, and the step of determining the set range is terminated; wherein the second set length is greater than the first set length.

[0021] In one embodiment, determining the depth information of the original image includes:

[0022] Depth information of the original image is determined based on the original image and the optical flow mapping relationship.

[0023] According to a second aspect of an embodiment of the present disclosure, there is provided an image transformation apparatus, comprising:

[0024] An acquisition module is used to obtain the original image and the optical flow mapping relationship between the original image and the target image;

[0025] The determination module is configured to determine the depth information of the original image; and determine the target image based on the original image, the optical flow mapping relationship, and the depth information of the original image.

[0026] In one embodiment, the optical flow mapping relationship is a forward optical flow mapping relationship;

[0027] The determining module is configured to:

[0028] Based on the forward optical flow mapping relationship, each original pixel point in the original image is mapped to the target image to obtain a mapping pixel point; among the mapping pixel points, the mapping pixel points located within the set range of the specified target pixel point position are determined; based on the depth information, among the mapping pixel points within the set range, the unobstructed mapping pixel points are screened out, and the target image is determined based on the unobstructed mapping pixel points.

[0029] In one embodiment, the determining module is configured to:

[0030] Based on the depth information, an occlusion relationship between mapping pixel points within a set range is determined; based on the occlusion relationship, a weight of each mapping pixel point is determined; wherein, a first mapping pixel point occludes a second mapping pixel point, and the weight of the first mapping pixel point is greater than the weight of the second mapping pixel point; and mapping pixel points whose weights of the mapping pixel points are greater than a weight threshold are determined as unoccluded mapping pixel points.

[0031] In one embodiment, the determining module is configured to:

[0032] Based on the distance between the unobstructed mapping pixel points, determine the weight of the distance from the position of each mapping pixel point in the unobstructed mapping pixel points to the designated pixel position of the target image; wherein, the distance from the first mapping pixel point to the designated pixel position of the target image is less than the distance from the second mapping pixel point to the designated pixel position of the target image, and the weight of the first mapping pixel point is greater than the weight of the second mapping pixel point; based on the weight from the position of each mapping pixel point to the designated pixel position, perform an interpolation operation on the unobstructed mapping pixel points to obtain the target pixel point corresponding to the designated pixel position; repeat the above process until the target pixel points of all pixel positions in the target image are obtained to obtain the target image.

[0033] In one embodiment, the optical flow mapping relationship is a backward optical flow mapping relationship;

[0034] The determining module is configured to:

[0035] Based on the backward optical flow mapping relationship, the optical flow of the specified target pixel point position in the target image is mapped to the original image; among the original pixel points in the original image, the original pixel points located within the set range of the position mapped by the specified target pixel point position are determined; based on the original pixel points within the set range, the target image is determined.

[0036] In one embodiment, the determination module is configured to:

[0037] Based on the distances between the original pixel points, the weight of the distance from the position of each original pixel point in the original pixel points to the mapped position of the specified target pixel point is determined; wherein, the distance from the first original pixel point to the mapped position of the target pixel point is smaller than the distance from the second original pixel point to the mapped position of the target pixel point, and the weight of the first original pixel point is larger than the weight of the second original pixel point; based on the weight from the position of each original pixel point to the specified pixel position, the original pixel points are interpolated to obtain the target pixel point corresponding to the specified pixel position; the above process is repeated until the target pixel points of all pixel positions in the target image are obtained to obtain the target image.

[0038] In one embodiment, the determining module is further configured to:

[0039] A set range is determined based on a first set length and a designated target pixel position; if the original pixel exists within the set range, the step of determining the set range is terminated; if the original pixel does not exist within the set range, the set range is re-determined based on a second set length and a designated target pixel position, and it is determined whether the original pixel exists within the re-determined set range; the step of determining the set range is repeated based on different set lengths until the original pixel exists within the re-determined set range, and the step of determining the set range is terminated; wherein the second set length is greater than the first set length.

[0040] In one embodiment, the determining module is configured to:

[0041] Depth information of the original image is determined based on the original image and the optical flow mapping relationship.

[0042] According to a third aspect of an embodiment of the present disclosure, there is provided an image transformation apparatus, comprising:

[0043] A processor; a memory for storing processor-executable instructions; wherein the processor is configured to: execute the image transformation method described in the first aspect or any one embodiment of the first aspect.

[0044] According to the fourth aspect of the embodiment of the present disclosure, a non-temporary computer-readable storage medium is provided. When the instructions in the storage medium are executed by the processor of the mobile terminal, the terminal is enabled to execute the image transformation method described in the first aspect or any one of the embodiments of the first aspect.

[0045] The technical solution provided by the embodiments of the present disclosure may include the following beneficial effects: through the optical flow mapping relationship between the original image and the target image, the target pixel points after the original pixel points are mapped can be obtained, and further based on the depth information of the original image, the position distribution relationship between the target pixel points can be determined, and then the evenly distributed target pixel points can be obtained based on the position distribution relationship.

[0046] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0048] Figure 1 The figure is a flowchart of an image transformation method according to an exemplary embodiment.

[0049] Figure 2 The present invention is a flowchart showing a forward optical flow mapping relationship in an image transformation method according to an exemplary embodiment.

[0050] Figure 3 The present invention is a flowchart of screening pixel points according to a forward optical flow mapping relationship in an image transformation method according to an exemplary embodiment.

[0051] Figure 4 The present invention is a flowchart of determining a target image by using forward optical flow mapping in an image transformation method according to an exemplary embodiment.

[0052] Figure 5 The present invention is a flowchart showing a backward optical flow mapping relationship in an image transformation method according to an exemplary embodiment.

[0053] Figure 6 The present invention is a flowchart of determining a target image by using a backward optical flow mapping relationship in an image transformation method according to an exemplary embodiment.

[0054] Figure 7 The present invention is a flowchart of determining a setting range in an image transformation method according to an exemplary embodiment.

[0055] Figure 8 The figure is a block diagram of an image conversion device according to an exemplary embodiment.

[0056] Figure 9 An electronic device is shown according to an exemplary embodiment. DETAILED DESCRIPTION

[0057] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, but not all of the embodiments.

[0058] In recent years, significant progress has been made in AI-based research on computer vision, deep learning, machine learning, image processing, and image recognition. Artificial Intelligence (AI) is an emerging science and technology that studies and develops theories, methods, technologies, and application systems for simulating and extending human intelligence. AI is a comprehensive discipline encompassing numerous technologies, including chips, big data, cloud computing, the Internet of Things, distributed storage, deep learning, machine learning, and neural networks. Computer vision, a key branch of AI, specifically enables machines to understand the world. Computer vision technologies typically include face recognition, liveness detection, fingerprint recognition and anti-counterfeiting verification, biometric recognition, face detection, pedestrian detection, object detection, pedestrian recognition, image processing, image recognition, image semantic understanding, image retrieval, text recognition, video processing, video content recognition, behavior recognition, 3D reconstruction, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), computational photography, and robotic navigation and positioning. With the research and advancement of artificial intelligence technology, this technology has been applied in many fields, such as security, urban management, traffic management, building management, park management, facial access, facial attendance, logistics management, warehouse management, robots, intelligent marketing, computational photography, mobile phone imaging, cloud services, smart homes, wearable devices, unmanned driving, autonomous driving, smart medical care, facial payment, facial unlocking, fingerprint unlocking, identity verification, smart screens, smart TVs, cameras, mobile Internet, live streaming, beauty, makeup, medical beauty, smart temperature measurement and other fields.

[0059] Image transformation involves mapping the pixels of an image based on specific rules to obtain the desired image. The mapping of pixels can be based on image coordinate mapping, i.e., creating a mapping relationship between the original image and the target image. The mapping relationship between the pixels of an original image and the pixels of a target image includes two types of mapping relationships. One type of mapping relationship is a forward mapping relationship, which can also be understood as forward mapping by calculating the coordinate position of any pixel of the original image in the mapped target image. The other type of mapping relationship is a backward mapping relationship, which can also be understood as backward mapping by calculating the coordinate position of any pixel of the transformed target image in the original image.

[0060] In forward mapping, if the pixels of the original image are unevenly distributed after being mapped to the target image, multiple pixels in the original image will be mapped to similar locations in the target image, resulting in "occlusion." Furthermore, some areas of the target image may not have any pixels mapped, resulting in "black holes."

[0061] Backward mapping is based on the pixel locations of the target image, mapping them to the original image, and determining the target pixel in the original image. This effectively assumes that the target pixel is derived from a single pixel in the original image or from an interpolated pixel. Furthermore, due to factors such as diffuse reflection, the color of a single point is determined by multiple light sources and surface normals. Therefore, determining the target pixel under this assumption has certain limitations.

[0062] Based on the above technical problems, the present disclosure proposes an image transformation method, which maps the pixels in the original image to the target image based on the optical flow mapping method, and maps the target pixels based on the depth information of the pixels, so as to fully consider the position distribution relationship between the pixels, thereby avoiding mapping defects such as "occlusion" and "black holes" in the target image caused by the uneven distribution of the pixels mapped from the original image.

[0063] Among them, the optical flow method is a change in the time domain. Based on the correlation between adjacent frames of the image, the corresponding relationship between the previous frame and the current frame is determined, thereby calculating the motion information of the object between adjacent frames. Taking the optical flow as the mapping relationship between the pixels in the original image and the pixels in the target image, it is possible to remap from the original image to the target image, fully considering the source of each target pixel. Based on the depth information of the pixel, the target pixel is mapped, which can reasonably utilize the information of the original image. Therefore, the image transformation method provided by the embodiment of the present disclosure not only makes the backward mapped pixel points unlimited, but also enables the target image after forward mapping to avoid problems such as "occlusion" and "black holes".

[0064] Embodiments of the Present Disclosure Hereinafter, the image transformation method provided by the present disclosure will be described with reference to the accompanying drawings.

[0065] Figure 1 FIG. 1 is a flow chart showing an image transformation method according to an exemplary embodiment. Figure 1 As shown, the image transformation method includes the following steps.

[0066] In step S11, an original image and an optical flow mapping relationship between the original image and a target image are obtained.

[0067] In the disclosed embodiment, acquiring an original image includes acquiring each original pixel in the original image and the position of each original pixel in the original image, where the position may be a pixel coordinate position. A target image is determined, and an optical flow mapping relationship between the original image and the target image is acquired, thereby determining the position of the target pixel.

[0068] In step S12 , the depth information of the original image is determined.

[0069] In the disclosed embodiments, depth information is obtained between each original pixel based on the original pixel position of the original image. This depth information is used to represent the distance between a point in the scene and the camera. In other words, depth information can be understood as the distance between the pixel and the capturing device. The capturing device can be a camera, a camcorder, or other device.

[0070] In step S13, a target image is determined based on the original image, the optical flow mapping relationship, and the depth information of the original image.

[0071] In the disclosed embodiments, based on each original pixel in the original image and the optical flow mapping relationship between the original image and the target image, depth information between the original pixels can be obtained through a deep learning network and calculation. Using the depth information and the original pixels, each target pixel is obtained, thereby obtaining the target image. The depth information can be relative depth information between the original pixels.

[0072] The image transformation method provided in the embodiments of the present disclosure uses an optical flow mapping relationship as the mapping relationship between original pixels and target pixels. This method can obtain the positions of the original pixels mapped to the mapped pixels of the target image. Based on the positions of the mapped pixels and their depth information, the occlusion relationship of the mapped pixels can be determined, thereby accurately obtaining the mapped pixels required to calculate the target image. The method can also obtain the mapping positions of the target pixel positions mapped to the original image, allowing it to consider more original pixels and thus obtain more accurate target pixels.

[0073] In the disclosed embodiments, the optical flow mapping relationship between the original image and the target image can be a forward optical flow mapping relationship, that is, mapping the original pixel points to the target image, and obtaining the mapping pixel points of the original pixel points to the target image. The optical flow mapping relationship between the original image and the target image can also be a backward optical flow mapping relationship, that is, mapping the position of the target pixel point in the target image to the original image, and obtaining the mapping position of the target pixel point to the original image.

[0074] The embodiment of this disclosure first describes how to determine the target image based on the mapped pixel points using the forward optical flow mapping relationship. Figure 2.

[0075] Figure 2 FIG. 1 is a flowchart illustrating a forward optical flow mapping relationship in an image transformation method according to an exemplary embodiment. Figure 2 As shown, the image transformation method includes the following steps.

[0076] In step S21, based on the forward optical flow mapping relationship, each original pixel in the original image is mapped to the target image to obtain a mapped pixel.

[0077] In the disclosed embodiment, the optical flow mapping of the original pixel to the mapped pixel in the target image can be obtained based on the forward optical flow mapping relationship. The position of the mapped pixel in the target image can also be calculated based on the position of the original pixel through the forward optical flow mapping relationship.

[0078] In step S22 , among the mapped pixels, the mapped pixels that are within the set range of the designated target pixel position are determined.

[0079] It should be understood that since the mapped pixel points are obtained by mapping the original pixel points to the target image through the forward optical flow mapping relationship, the positions of the mapped pixel points are usually non-integer pixels. The target pixel point is the target theoretical pixel position in the target image mapped by the forward optical flow mapping of the original pixel points in the original image, which is generally an integer pixel point.

[0080] In the embodiments of the present disclosure, the position of a designated target pixel is determined, and the positions of the mapped pixels within the position range are determined. The set range involved in the embodiments of the present disclosure can be determined based on the position of the target pixel and the position of the mapped pixel in a two-dimensional coordinate system. For example, in the embodiments of the present disclosure, the two-dimensional coordinate range can be determined based on a preset two-dimensional coordinate distance with the position of the target pixel as the coordinate origin. Of course, the set range can also be determined in other ways, which are not specifically limited here.

[0081] In step S23 , based on the depth information, unobstructed mapping pixel points are screened out from the mapping pixel points within the set range, and a target image is determined based on the unobstructed mapping pixel points.

[0082] In an embodiment of the present disclosure, depth information of mapped pixels within a set range is obtained, and unobstructed mapped pixels within the set range are screened out using the obtained depth information. The screened unobstructed mapped pixels include a plurality of unobstructed mapped pixels. Furthermore, the designated target pixel is determined by calculation from the plurality of unobstructed mapped pixels.

[0083] By determining the target pixel points according to the forward optical flow mapping relationship in the above embodiment, all target pixel points required in the target image are obtained, thereby determining the target image.

[0084] Through the image transformation method provided in the embodiments of the present invention, for each target pixel point, the mapped pixel points mapped from the original pixel points around it are considered. By screening the occluded mapped pixel points and further determining the target pixel points based on the unoccluded mapped pixel points, problems such as "occlusion" and "black holes" can be solved.

[0085] The following embodiments of the present disclosure will illustrate how to filter out unobstructed mapped pixels from mapped pixels within a set range based on depth information. The implementation steps can be found in Figure 3 .

[0086] Figure 3 FIG. 1 is a flowchart of selecting pixel points in a forward optical flow mapping relationship in an image transformation method according to an exemplary embodiment. Figure 3 As shown, the image transformation method includes the following steps.

[0087] In step S31 , based on the depth information, the occlusion relationship between the mapped pixels within the set range is determined.

[0088] In an embodiment of the present disclosure, the depth information may be based on relative depth information between mapped pixels within a set range, and the occlusion relationship between the mapped pixels within the set range is determined based on the relative depth information.

[0089] In step S32, the weight of each mapped pixel is determined based on the occlusion relationship.

[0090] In the embodiment of the present disclosure, if a first mapped pixel occludes a second mapped pixel, the weight of the first mapped pixel is greater than the weight of the second mapped pixel. The determined weights can be used to subsequently filter out unoccluded mapped pixels.

[0091] For example, taking mapping pixel point A, mapping pixel point B, and mapping pixel point C as an example, if mapping pixel point A and mapping pixel point B occlude mapping pixel point C, then the weight of mapping pixel point C is determined to be the smallest, and then based on the relative depth information between mapping pixel point A and mapping pixel point B, the weights of mapping pixel point A and mapping pixel point B are determined. If mapping pixel point A is before mapping pixel point B, the weight of mapping pixel point A is the largest.

[0092] In step S33 , the mapping pixel points whose weights are greater than the weight threshold are determined as unblocked mapping pixel points.

[0093] In the disclosed embodiment, the weight of each mapped pixel within a set range is determined, and a preset weight threshold is determined. Mapped pixels whose weights are greater than the weight threshold are determined as unobstructed mapped pixels, and mapped pixels whose weights are less than or equal to the weight threshold are determined as obstructed mapped pixels. In this way, a plurality of unobstructed mapped pixels within the set range can be obtained.

[0094] For example, continuing with the above embodiment, among mapping pixel point A, mapping pixel point B, and mapping pixel point C, mapping pixel point A can be screened out as an unobstructed mapping pixel point, or mapping pixel point A and mapping pixel point B can be screened out as unobstructed mapping pixel points.

[0095] Through the image transformation method provided in the embodiment of the present disclosure, the occluded mapping pixel points can be filtered out first, that is, it can be understood as removing the occluded mapping pixel points, and determining the target pixel points among the unoccluded mapping pixel points, thereby solving the occlusion problem after pixel mapping, and the determined target pixel points are more accurate.

[0096] The following embodiments of the present disclosure will illustrate how to determine the target image based on the unobstructed mapped pixels. The implementation steps can be found in Figure 4 .

[0097] Figure 4 FIG. 1 is a flowchart illustrating a method for image transformation using forward optical flow mapping to determine a target image according to an exemplary embodiment. Figure 4 As shown, the image transformation method includes the following steps.

[0098] In step S41 , based on the distances between the unobstructed mapped pixels, a weight of the distance between the position of each unobstructed mapped pixel and the position of a designated pixel in the target image is determined.

[0099] In the disclosed embodiment, based on the positions of multiple unobstructed mapped pixels, the distance between the unobstructed mapped pixels can be determined, thereby determining the distance between each mapped pixel and a specified pixel position in the target image. The weight of each mapped pixel is determined based on the different distances.

[0100] Exemplarily, if the distance between the first mapped pixel point and the designated pixel position of the target image is smaller than the distance between the second mapped pixel point and the designated pixel position of the target image, the weight of the first mapped pixel point is greater than the weight of the second mapped pixel point.

[0101] In step S42 , based on the weight of each mapped pixel position to the designated pixel position, an interpolation operation is performed on the unobstructed mapped pixel points to obtain a target pixel point corresponding to the designated pixel position.

[0102] In step S43, the above steps are repeated until target pixel points at all pixel positions in the target image are obtained to obtain the target image.

[0103] In the disclosed embodiment, an interpolation operation is performed on the unobstructed mapped pixels based on the weight of each unobstructed mapped pixel to determine a target pixel to be placed at the target pixel's location. The interpolation operation can be a nearest neighbor interpolation operation, a linear interpolation operation, or the like. The above steps are repeated to obtain target pixels at all pixel locations in the target image, thereby obtaining the target image.

[0104] In the embodiment of the present disclosure, the position of the target pixel point is determined by optical flow mapping, and the positional relationship between the mapping pixels is determined based on the position of the target pixel point and the depth information of the mapping pixel point, that is, the occlusion relationship between the mapping pixels is determined by a single weighting method, thereby screening out the unobstructed mapping pixels. Based on the unobstructed mapping pixels, the target pixel point is determined by interpolation operation (second weighting). In other words, in the present disclosure, the mapping pixel points are processed by two weighting processes to determine the target pixel point of the position of the target pixel point. Through the embodiment of the present disclosure, the positional distribution relationship between the pixels can be fully considered. By removing the occluded original pixel points based on the depth information, mapping defects such as "occlusion" and "black holes" of the mapping pixels in the target image caused by the forward optical flow mapping relationship can be avoided.

[0105] In the embodiment of the present disclosure, if the backward mapping method is adopted, the target pixel position of the target image is mapped to the original image based on the backward optical flow mapping relationship, and the mapping position of the target pixel position in the original image is determined. Based on the mapping position of the target pixel position and the preset range, the original pixel point located within the set range of the mapped position of the target pixel point position is determined. Subsequently, based on the depth information of the original pixel point in the original image, the target pixel point is determined in the mapped pixel point. The embodiment of the present disclosure describes the use of the backward optical flow mapping relationship to determine the target image based on the original image, the optical flow mapping relationship and the depth information of the original image. Its implementation steps can be found in Figure 5 .

[0106] Figure 5 FIG. 1 is a flowchart showing a backward optical flow mapping relationship in an image transformation method according to an exemplary embodiment. Figure 5 As shown, the image transformation method includes the following steps.

[0107] In step S51, based on the backward optical flow mapping relationship, the optical flow of the designated target pixel position in the target image is mapped to the original image.

[0108] In the disclosed embodiments, the mapping position of the target pixel position in the transformed target image to the original image is calculated based on the backward optical flow mapping relationship. That is, the target pixel position is optically mapped to the original image to determine the mapping position of the target pixel position. The mapping position of the target pixel position is generally a non-integer pixel point.

[0109] In step S52 , original pixel points located within a set range of the position mapped to the designated target pixel point position are determined among the original pixel points of the original image.

[0110] In an embodiment of the present disclosure, the mapping position of a specified target pixel is determined, and the original pixel points located within the position range are determined in the original image. Among them, the set range involved in the embodiment of the present disclosure can be determined based on the mapping position of the target pixel point and the position of the original pixel point in the two-dimensional coordinate system. As mentioned above, in the embodiment of the present disclosure, the two-dimensional coordinate range can be determined based on a preset two-dimensional coordinate distance with the mapping position of the target pixel point as the coordinate origin. Of course, the set range can also be determined in other ways, which are not specifically limited here.

[0111] In step S53, a target image is determined based on the original pixel points within the set range.

[0112] In an embodiment of the present disclosure, the positions of original pixels within a set range are obtained, and the depth information thereof is determined based on the positions of the original pixels. Based on the obtained depth information, unobstructed original pixels are screened out from the original pixels within the set range. The screened unobstructed original pixels include a plurality of original pixels. Furthermore, among the plurality of unobstructed original pixels, a target pixel at the position of the designated target pixel is determined by interpolation calculation.

[0113] By determining the target pixel points according to the backward optical flow mapping relationship in the above embodiment, all target pixel points required in the target image are obtained, thereby determining the target image.

[0114] Through the image transformation method provided by the embodiment of the present disclosure, in the process of backward mapping of pixels, the position information of each target pixel point in the original image is fully considered, the information of the original image can be reasonably utilized, and the original pixel points mapped around it are fully considered, breaking through the limitations of backward mapping of pixel endpoints.

[0115] In the embodiment of the present disclosure, when the backward optical flow mapping relationship is adopted, the target pixel point position mapped to the original image is determined as the target pixel point mapping position. Since the original pixel points existing in the original image are all integer pixel points, when determining the original pixel point, the target pixel point mapping position setting range can be determined based on the position of the original pixel point in the original image and the set length in the horizontal direction and / or the set length in the vertical direction in the preset two-dimensional coordinate system. Among them, the following implementation method will illustrate how to determine the target image based on the original pixel points within the set range. Its implementation steps can be found in Figure 6 .

[0116] Figure 6 FIG. 1 is a flowchart illustrating a method for image transformation using a backward optical flow mapping relationship to determine a target image according to an exemplary embodiment. Figure 6 As shown, the image transformation method includes the following steps.

[0117] In step S61 , based on the distance between the original pixels, the weight of the distance between the position of each original pixel and the mapped position of the designated target pixel is determined.

[0118] In the disclosed embodiment, based on the positions of multiple unobstructed original pixels, the distance between the unobstructed original pixels can be determined, thereby determining the distance between the position of each original pixel and the position of the target pixel. The weight of each original pixel is determined based on the different distances.

[0119] Exemplarily, the distance between the first original pixel and the target pixel is smaller than the distance between the second original pixel and the target pixel, and the weight of the first original pixel is greater than the weight of the second original pixel.

[0120] In step S62 , based on the weight of each original pixel position to the designated pixel position, an interpolation operation is performed on the original pixel points to obtain a target pixel point corresponding to the designated pixel position.

[0121] In step S63, the above steps are repeated until target pixel points at all pixel positions in the target image are obtained to obtain the target image.

[0122] In the disclosed embodiment, an interpolation operation is performed on the unobstructed original pixels based on the weight of each unobstructed original pixel to determine a target pixel to be placed at the target pixel location. The interpolation operation can be a nearest neighbor interpolation operation, a linear interpolation operation, or the like. The above steps are repeated to obtain target pixel locations for all pixel locations in the target image, thereby obtaining the target image.

[0123] The disclosed embodiment, using the backward optical flow mapping relationship, can more accurately determine the target pixel through the deocclusion step, fully consider the surrounding mapped pixels, and provide a more optimized solution that breaks through the limitations of backward mapping of pixel endpoints.

[0124] In the embodiment of the present disclosure, whether based on the forward optical flow mapping relationship or the backward optical flow mapping relationship, it is necessary to obtain pixel points within a set range to determine the target pixel point. Figure 7 As shown, the setting range can be determined by the following implementation steps.

[0125] Figure 7 FIG. 1 is a flow chart showing a method for determining a set range in an image transformation according to an exemplary embodiment. Figure 7 As shown, the image transformation method includes the following steps.

[0126] In step S71 , a setting range is determined based on a first setting length and a designated target pixel position.

[0127] In the embodiment of the present disclosure, the first set length can be set to an empirical value according to actual needs. When the position of the designated target pixel point is determined, the set range can be determined based on the preset first set length.

[0128] In step S72 , if the original pixel point exists within the set range, the step of determining the set range is terminated.

[0129] In the embodiment of the present disclosure, the original pixel points within the set range can be interpolated to determine whether there are original pixel points mapped to the target pixel point mapping position. That is, if the original pixel point mapping position after the interpolation processing is the target pixel point mapping position, it is determined that there are original pixel points mapped to the target pixel point mapping position. If the pixel point mapping position after the interpolation processing is not the target pixel point mapping position, it is determined that there are no original pixel points mapped to the target pixel point mapping position. If there are original pixel points within the set range, the step of determining the set range is terminated, and as described above, the occluded original pixel points are removed within the range, and the step of obtaining the target image is executed.

[0130] In step S73 , if the original pixel does not exist within the set range, the set range is re-determined based on the second set length and the designated target pixel position, and it is determined whether the original pixel exists within the re-determined set range.

[0131] If the original pixel does not exist within the set range, a second set length is determined. Based on the second set length and the specified target pixel location, the set range is re-determined, and it is again determined whether the original pixel exists within the re-determined set range. The second set length can be understood as the length used to reset the set range. The second set length is greater than the first set length.

[0132] In step S74, the step of determining the set range is repeatedly executed based on different set lengths until the original pixel point exists within the re-determined set range, and the step of determining the set range is terminated.

[0133] In the embodiment of the present disclosure, the steps of determining the set range and whether there are original pixel points within the set range that can determine the target pixel point can be repeated until there are original pixel points within the set range and the target pixel point can be determined based on the original pixel points, and the steps of determining the set range are ended.

[0134] In an example of an embodiment of the present disclosure, it is assumed that the first set length is N, the second set length is M, and M is greater than N, wherein the values ​​of N and M can be determined according to actual conditions or set to fixed values.

[0135] If, based on N and the position of the target pixel point, it is determined that there is an original (mapped) pixel point mapped to the target pixel point position within the set range, then the original (mapped) pixel point mapping position within the N range is determined as the original (mapped) pixel point mapping position within the set range of the target pixel point mapping position. If, based on N and the target pixel point mapping position, it is determined that there is no original (mapped) pixel point of the target pixel point mapping position within the set range, then, based on M and the position of the target pixel point, it is re-determined whether there is an original (mapped) pixel point of the target pixel point mapping position within the set range. If, based on M and the position of the target pixel point, it is determined that there is an original (mapped) pixel point of the target pixel point position within the set range, then the original (mapped) pixel point mapping position within M range is determined as the original (mapped) pixel point mapping position within the set range of the target pixel point position. If, based on M and the position of the target pixel point, it is determined that there is no original (mapped) pixel point of the target pixel point position within the set range, then, the set length is re-determined and the above process is repeated until there is an original (mapped) pixel point of the target pixel point position within the set range.

[0136] In the embodiment of the present disclosure, the depth information of the original image can be determined based on the original image and the optical flow mapping relationship.

[0137] The depth information may be relative depth information of original pixels in the original image.

[0138] In the embodiment of the present disclosure, the occlusion relationship between original (mapped) pixels is determined based on relative depth information, without the need to obtain accurate actual depth information of each mapped pixel, thereby reducing the amount of calculation and complexity.

[0139] Based on the same / similar concept, an embodiment of the present disclosure further provides an image transformation device.

[0140] It is understandable that the image transformation device provided by the embodiment of the present disclosure includes hardware structures and / or software modules corresponding to the execution of each function in order to realize the above functions. In combination with the units and algorithm steps of each example disclosed in the embodiment of the present disclosure, the embodiment of the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to realize the described functions for each specific application, but such implementation should not be considered to exceed the scope of the technical solution of the embodiment of the present disclosure.

[0141] Figure 8 FIG. 1 is a block diagram of an image conversion device according to an exemplary embodiment. Figure 8 The image transformation device 100 includes an acquisition module 101 and a determination module 102.

[0142] The acquisition module 101 is used to acquire an original image and an optical flow mapping relationship between the original image and a target image. The determination module 102 is used to determine the depth information of the original image and determine the target image based on the original image, the optical flow mapping relationship, and the depth information of the original image.

[0143] In the embodiment of the present disclosure, the optical flow mapping relationship is a forward optical flow mapping relationship.

[0144] Determination module 102 is configured to map each original pixel in the original image to a target image based on the forward optical flow mapping relationship to obtain mapped pixels. Among the mapped pixels, mapping pixels within a set range of the specified target pixel position are determined. Based on depth information, unobstructed mapping pixels are screened from the mapped pixels within the set range, and the target image is determined based on the unobstructed mapping pixels.

[0145] In the disclosed embodiment, the determination module 102 is configured to determine an occlusion relationship between mapped pixels within a set range based on depth information. A weight for each mapped pixel is determined based on the occlusion relationship. Where a first mapped pixel occludes a second mapped pixel, the weight of the first mapped pixel is greater than the weight of the second mapped pixel. Mapped pixels whose weights are greater than a weight threshold are determined as unoccluded mapped pixels.

[0146] In the embodiment of the present disclosure, the determination module 102 is used to determine the weight of the distance from the position of each unobstructed mapped pixel point to the designated pixel position of the target image based on the distance between the unobstructed mapped pixel points. Wherein, the distance from the first mapped pixel point to the designated pixel position of the target image is less than the distance from the second mapped pixel point to the designated pixel position of the target image, and the weight of the first mapped pixel point is greater than the weight of the second mapped pixel point. Based on the weight of the position of each mapped pixel point to the designated pixel position, an interpolation operation is performed on the unobstructed mapped pixel points to obtain the target pixel point corresponding to the designated pixel position. The above process is repeated until the target pixel points of all pixel positions in the target image are obtained to obtain the target image.

[0147] In the embodiment of the present disclosure, the optical flow mapping relationship is a backward optical flow mapping relationship.

[0148] Determination module 102 is configured to map the optical flow of a specified target pixel position in the target image to the original image based on the backward optical flow mapping relationship. Determination module 102 determines, from among the original pixels in the original image, the original pixels that are within a set range of the position mapped to the specified target pixel position. The target image is determined based on the original pixels within the set range.

[0149] In the embodiment of the present disclosure, the determination module 102 is used to determine the weight of the distance between the position of each original pixel point in the original pixel points and the position mapped to the specified target pixel point based on the distance between the original pixel points. Wherein, the distance between the first original pixel point and the position mapped to the target pixel point is less than the distance between the second original pixel point and the position mapped to the target pixel point, and the weight of the first original pixel point is greater than the weight of the second original pixel point. Based on the weight of the position of each original pixel point to the specified pixel position, the original pixel points are interpolated to obtain the target pixel point corresponding to the specified pixel position. The above process is repeated until the target pixel points of all pixel positions in the target image are obtained to obtain the target image.

[0150] In the embodiment of the present disclosure, the determination module 102 is further used to determine the set range based on the first set length and the specified target pixel position. If the original pixel exists within the set range, the step of determining the set range is terminated. If the original pixel does not exist within the set range, the set range is re-determined based on the second set length and the specified target pixel position, and it is determined whether the original pixel exists within the re-determined set range. The step of determining the set range is repeated based on different set lengths until the original pixel exists within the re-determined set range, and the step of determining the set range is terminated. The second set length is greater than the first set length.

[0151] In the embodiment of the present disclosure, the determination module 102 is configured to determine depth information of the original image based on the original image and the optical flow mapping relationship.

[0152] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0153] like Figure 9 As shown, one embodiment of the present disclosure provides an electronic device 200. The electronic device 200 includes a memory 201, a processor 202, and an input / output (I / O) interface 203. The memory 201 is used to store instructions. The processor 202 is used to call the instructions stored in the memory 201 to execute the image transformation method of the embodiment of the present disclosure. The processor 202 is connected to the memory 201 and the I / O interface 203 respectively, for example, through a bus system and / or other forms of connection mechanisms (not shown). The memory 201 can be used to store programs and data, including the program of the image transformation method involved in the embodiment of the present disclosure, and the processor 202 executes various functional applications and data processing of the electronic device 200 by running the program stored in the memory 201.

[0154] In the embodiment of the present disclosure, the processor 202 can be implemented in at least one hardware form of a digital signal processor (DSP), a field programmable gate array (FPGA), or a programmable logic array (PLA). The processor 202 can be a central processing unit (CPU) or one or a combination of other forms of processing units with data processing capabilities and / or instruction execution capabilities.

[0155] The memory 201 in the embodiment of the present disclosure may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD).

[0156] In the embodiment of the present disclosure, the I / O interface 203 may be used to receive input commands (e.g., digital or character information, and to generate key signal input related to user settings and function control of the electronic device 200), and may also output various information (e.g., images or sounds) to the outside. In the embodiment of the present disclosure, the I / O interface 203 may include one or more of a physical keyboard, function keys (e.g., volume control keys, power keys, etc.), a mouse, a joystick, a trackball, a microphone, a speaker, and a touch panel.

[0157] It should be understood that although operations are described in a particular order in the drawings in the embodiments of the present disclosure, this should not be construed as requiring that these operations be performed in the particular order shown or in a serial order, or that all of the operations shown be performed to obtain the desired results. In certain circumstances, multitasking and parallel processing may be advantageous.

[0158] The methods and apparatuses of the embodiments of the present disclosure can be implemented using standard programming techniques, utilizing rule-based logic or other logic to implement the various method steps. It should also be noted that the terms "apparatus" and "module" as used herein and in the claims are intended to include implementations using one or more lines of software code and / or hardware implementations and / or devices for receiving input.

[0159] Any steps, operations or procedures described herein may be performed or implemented using one or more hardware or software modules, either alone or in combination with other devices. In one embodiment, the software modules are implemented using a computer program product comprising a computer-readable medium containing computer program code, which can be executed by a computer processor to perform any or all of the steps, operations or procedures described.

[0160] The foregoing description of embodiments of the present disclosure has been provided for purposes of illustration and description. The foregoing description is not intended to be exhaustive or to limit the disclosure to the precise form disclosed, and various variations and modifications are possible in light of the foregoing teachings or from practice of the disclosure. These embodiments have been chosen and described to illustrate the principles of the disclosure and its practical application, so as to enable those skilled in the art to utilize the disclosure in various embodiments and with various modifications as appropriate for the particular use contemplated.

Claims

1. An image transformation method, characterized in that: The method comprises: Obtain the original image and the optical flow mapping relationship between the original image and the target image; Determining depth information of the original image; Determining a target image based on the original image, the optical flow mapping relationship, and depth information of the original image; If the optical flow mapping relationship is a backward optical flow mapping relationship, determining the target image based on the original image, the optical flow mapping relationship, and the depth information of the original image includes: Based on the backward optical flow mapping relationship, mapping the optical flow of the specified target pixel position in the target image to the original image; Determine, among the original pixels of the original image, the original pixels that are within a set range of the position mapped to the designated target pixel position; Based on the depth information, unobstructed original pixel points are screened out from original pixel points within a set range, and a target image is determined based on the unobstructed mapped pixel points.

2. The image transformation method according to claim 1, wherein: If the optical flow mapping relationship is a forward optical flow mapping relationship, determining the target image based on the original image, the optical flow mapping relationship, and the depth information of the original image includes: Based on the forward optical flow mapping relationship, each original pixel in the original image is mapped to the target image to obtain a mapped pixel; Among the mapped pixels, determining the mapped pixels that are within a set range of the designated target pixel position; Based on the depth information, unobstructed mapping pixel points are screened out from the mapping pixel points within a set range, and a target image is determined based on the unobstructed mapping pixel points.

3. The image transformation method according to claim 2, wherein: The step of selecting unobstructed mapped pixel points from the mapped pixel points within a set range based on the depth information includes: Determining an occlusion relationship between mapped pixels within a set range based on the depth information; Determining a weight of each mapped pixel based on the occlusion relationship; wherein a first mapped pixel occludes a second mapped pixel, and the weight of the first mapped pixel is greater than the weight of the second mapped pixel; The mapped pixel points whose weights are greater than a weight threshold are determined as unobstructed mapped pixel points.

4. The image transformation method according to claim 2, wherein: The determining of the target image based on the unobstructed mapped pixel points includes: Determining, based on the distances between the unobstructed mapped pixels, a weight for the distance between the position of each of the unobstructed mapped pixels and the designated pixel position of the target image; wherein, if the distance between a first mapped pixel point and the designated pixel position of the target image is smaller than the distance between a second mapped pixel point and the designated pixel position of the target image, the weight of the first mapped pixel point is greater than the weight of the second mapped pixel point; Based on the weight of the position of each mapped pixel point to the designated pixel position, interpolation operation is performed on the unobstructed mapped pixel points to obtain a target pixel point corresponding to the designated pixel position; Repeat the above steps until the target pixel points of all pixel positions in the target image are obtained to obtain the target image.

5. The image transformation method according to claim 1, wherein: Determining a target image based on original pixel points within the set range includes: Determining, based on the distances between the original pixels, a weight of a distance between the position of each original pixel and the position of a specified target pixel; wherein, if the distance between a first original pixel and the position of the target pixel is smaller than the distance between a second original pixel and the position of the target pixel, the weight of the first original pixel is greater than the weight of the second original pixel; Based on the weight of the position of each original pixel point to the designated pixel position, interpolation operation is performed on the original pixel point to obtain a target pixel point corresponding to the designated pixel position; Repeat the above steps until the target pixel points of all pixel positions in the target image are obtained to obtain the target image.

6. The image transformation method according to claim 2, wherein: The setting range is determined as follows: Determining a set range based on a first set length and a designated target pixel position; If the original pixel point exists within the set range, then the step of determining the set range is terminated; If the original pixel does not exist within the set range, redetermine the set range based on the second set length and the designated target pixel position, and determine whether the original pixel exists within the redetermined set range; Repeating the step of determining the set range based on different set lengths until the original pixel point exists within the newly determined set range, and then ending the step of determining the set range; Wherein, the second set length is greater than the first set length.

7. The image transformation method according to any one of claims 1 to 6, characterized in that: The determining the depth information of the original image includes: Depth information of the original image is determined based on the original image and the optical flow mapping relationship.

8. An image conversion device, characterized in that: The device comprises: An acquisition module is used to obtain the original image and the optical flow mapping relationship between the original image and the target image; a determination module, configured to determine the depth information of the original image; and determine a target image based on the original image, the optical flow mapping relationship, and the depth information of the original image; If the optical flow mapping relationship is a backward optical flow mapping relationship, the determination module is used to: map the optical flow of the specified target pixel position in the target image to the original image based on the backward optical flow mapping relationship; determine the original pixel points within the set range of the mapped position of the specified target pixel point among the original pixel points in the original image; based on the depth information, filter out the unobstructed original pixel points among the original pixel points within the set range, and determine the target image based on the unobstructed mapped pixel points.

9. An electronic device, wherein: The electronic device comprises: a memory for storing instructions; and A processor, configured to call the instructions stored in the memory to execute the image transformation method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing instructions, wherein when the instructions are executed by a processor, the image transformation method according to any one of claims 1 to 7 is executed.

Images