Image generation method and device, electronic equipment and computer readable medium

Abstract

The invention discloses an image generation method and device, electronic equipment and a computer readable medium, and the method comprises the steps: firstly obtaining at least two to-be-used images, so as to enable different to-be-used images to be used for describing the states of a target object at different visual angles, therefore, the at least two to-be-used images can represent the states of the target object under the at least two view angles; constructing a position mapping relation so that the position mapping relation is used for describing a corresponding relation between pixel positions in the at least two to-be-used images and pixel positions in the fisheye image; and finally, according to the position mapping relationship and the pixel values in the at least two to-be-used images, generating the fisheye image, so that the fisheye image can describe the states of the target object under the at least two view angles, thereby enabling the fisheye image to better describe the target object, and being beneficial to improving the obtaining effect of the fisheye image.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image generation method, device, electronic device, and computer-readable medium. Background Art

[0002] In some application scenarios (e.g., 3D object design, 3D object display, etc.), it may be necessary to obtain a fisheye image of an object (e.g., a house model, a building model, etc.). Therefore, how to obtain a fisheye image becomes a technical problem that needs to be solved urgently. Summary of the invention

[0003] In order to solve the above technical problems, the present application provides an image generation method, device, electronic device, and computer-readable medium.

[0004] In order to achieve the above objectives, the technical solutions provided by this application are as follows:

[0005] The present application provides an image generation method, the method comprising:

[0006] Acquire at least two images to be used, where different images to be used are used to describe states of the target object at different viewing angles;

[0007] Constructing a position mapping relationship, where the position mapping relationship is used to describe the correspondence between the pixel positions in the at least two images to be used and the pixel positions in the fisheye image;

[0008] The fisheye image is generated based on the position mapping relationship and the at least two images to be used. The pixel values ​​in the fisheye image are determined based on the pixel values ​​in the at least two images to be used. The fisheye image is used to describe the state of the target object under at least two viewing angles.

[0009] In a possible implementation manner, the position mapping relationship includes a first sub-mapping relationship and a second sub-mapping relationship;

[0010] The first sub-mapping relationship is used to describe the correspondence between the pixel position in the fisheye image and the three-dimensional coordinate position in the three-dimensional spherical surface;

[0011] The second sub-mapping relationship is used to describe the correspondence between the three-dimensional coordinate position in the three-dimensional spherical surface and the pixel position in the at least two images to be used.

[0012] In a possible implementation manner, the process of determining the second sub-mapping relationship includes:

[0013] For any three-dimensional coordinate position in the three-dimensional spherical surface, according to the preset mapping conditions of each of the images to be used, a target image corresponding to the three-dimensional coordinate position is determined from the at least two images to be used; the three-dimensional coordinate position satisfies the preset mapping conditions of the target image;

[0014] Determine a mapping pixel position corresponding to the three-dimensional coordinate position from the target image;

[0015] The second sub-mapping relationship is determined according to the correspondence relationship between the three-dimensional coordinate position and the mapping pixel position, and the second sub-mapping relationship includes the correspondence relationship between the three-dimensional coordinate position and the mapping pixel position.

[0016] In a possible implementation manner, for any image to be used among the at least two images to be used, the preset mapping condition of the image to be used is determined according to a capture viewing angle of the image to be used.

[0017] In a possible implementation manner, the target image is used to describe the state of the target object from a top surface perspective, and the preset mapping conditions of the target image are: the data value in the third dimension of the three-dimensional coordinate position is greater than 0, the absolute value of the data value in the third dimension of the three-dimensional coordinate position is not less than the absolute value of the data value in the first dimension of the three-dimensional coordinate position, and the absolute value of the data value in the third dimension of the three-dimensional coordinate position is not less than the absolute value of the data value in the second dimension of the three-dimensional coordinate position;

[0018] or,

[0019] The target image is used to describe the state of the target object from the bottom surface perspective, and the preset mapping conditions of the target image are: the data value in the third dimension of the three-dimensional coordinate position is less than 0, the absolute value of the data value in the third dimension of the three-dimensional coordinate position is not less than the absolute value of the data value in the first dimension of the three-dimensional coordinate position, and the absolute value of the data value in the third dimension of the three-dimensional coordinate position is not less than the absolute value of the data value in the second dimension of the three-dimensional coordinate position;

[0020] or,

[0021] The target image is used to describe the state of the target object in the frontal perspective, and the preset mapping conditions of the target image are: the data value in the first dimension of the three-dimensional coordinate position is greater than 0, the absolute value of the data value in the first dimension of the three-dimensional coordinate position is not less than the absolute value of the data value in the second dimension of the three-dimensional coordinate position, and the absolute value of the data value in the first dimension of the three-dimensional coordinate position is not less than the absolute value of the data value in the third dimension of the three-dimensional coordinate position;

[0022] or,

[0023] The target image is used to describe the state of the target object from a rear perspective, and the preset mapping conditions of the target image are: the data value in the first dimension of the three-dimensional coordinate position is less than 0, the absolute value of the data value in the first dimension of the three-dimensional coordinate position is not less than the absolute value of the data value in the second dimension of the three-dimensional coordinate position, and the absolute value of the data value in the first dimension of the three-dimensional coordinate position is not less than the absolute value of the data value in the third dimension of the three-dimensional coordinate position;

[0024] or,

[0025] The target image is used to describe the state of the target object in the right viewing angle, and the preset mapping conditions of the target image are: the data value in the second dimension of the three-dimensional coordinate position is greater than 0, the absolute value of the data value in the second dimension of the three-dimensional coordinate position is not less than the absolute value of the data value in the first dimension of the three-dimensional coordinate position, and the absolute value of the data value in the second dimension of the three-dimensional coordinate position is not less than the absolute value of the data value in the third dimension of the three-dimensional coordinate position;

[0026] or,

[0027] The target image is used to describe the state of the target object in the left viewing angle, and the preset mapping conditions of the target image are: the data value in the second dimension of the three-dimensional coordinate position is less than 0, the absolute value of the data value in the second dimension of the three-dimensional coordinate position is not less than the absolute value of the data value in the first dimension of the three-dimensional coordinate position, and the absolute value of the data value in the second dimension of the three-dimensional coordinate position is not less than the absolute value of the data value in the third dimension of the three-dimensional coordinate position.

[0028] In a possible implementation manner, determining the mapping pixel position corresponding to the three-dimensional coordinate position from the target image includes:

[0029] The mapping pixel position corresponding to the three-dimensional coordinate position is determined according to a preset mapping algorithm corresponding to the target image, wherein the preset mapping algorithm is determined according to the acquisition viewing angle of the target image.

[0030] In a possible implementation manner, the preset mapping algorithm corresponding to the target image includes:

[0031] Determining a data compression rate according to a data value in a target dimension in the three-dimensional coordinate position; the target dimension is determined according to a capture viewing angle of the target image;

[0032] The mapping pixel position corresponding to the three-dimensional coordinate position is determined according to the data compression rate and other two data values ​​in the three-dimensional coordinate position except the data value in the target dimension.

[0033] In a possible implementation manner, determining the mapping pixel position corresponding to the three-dimensional coordinate position according to the data compression rate and two other data values ​​in the three-dimensional coordinate position except the data value in the target dimension includes:

[0034] Determine a theoretical mapping position corresponding to the three-dimensional coordinate position according to the data compression rate and two other data values ​​in the three-dimensional coordinate position except the data value in the target dimension;

[0035] Determine at least one adjacent pixel position corresponding to the theoretical mapping position from the target image;

[0036] A mapping pixel position corresponding to the three-dimensional coordinate position is determined according to the at least one adjacent pixel position.

[0037] In a possible implementation manner, the mapped pixel position corresponding to the three-dimensional coordinate position includes the at least one adjacent pixel position;

[0038] The pixel value at the pixel position corresponding to the three-dimensional coordinate position in the fisheye image is obtained by weighted summing the pixel values ​​at the at least one adjacent pixel position in the target image;

[0039] For any adjacent pixel position, the weighted weight corresponding to the pixel value at the adjacent pixel position in the target image is determined according to the distance between the adjacent pixel position and the theoretical mapping position.

[0040] In a possible implementation manner, the at least one adjacent pixel position includes four adjacent pixel positions.

[0041] In a possible implementation manner, the at least two viewing angles include K viewing angles, where K is a positive integer;

[0042] The at least two images to be used include K images to be used, and the kth image to be used is obtained by capturing the target object according to the kth viewing angle, so that the kth image to be used is used to describe the state of the target object at the kth viewing angle, k is a positive integer, k≤K.

[0043] In a possible implementation manner, the kth image to be used is obtained by capturing the target object according to the kth viewing angle and the first wide angle;

[0044] The step of generating the fisheye image according to the position mapping relationship and the at least two images to be used includes:

[0045] Determine, according to the image size of the kth image to be used, image region description information corresponding to a second wide angle in the kth image to be used; the second wide angle is smaller than the first wide angle, k is a positive integer, k≤K;

[0046] Extracting pixel values ​​from the kth image to be used according to the image region description information corresponding to the second wide angle in the kth image to be used, to obtain the pixel value corresponding to the second wide angle in the kth image to be used, where k is a positive integer, k≤K;

[0047] The fisheye image is generated according to the position mapping relationship and the pixel values ​​corresponding to the second wide angle in the K images to be used.

[0048] In a possible implementation manner, the kth image to be used is obtained by capturing the target object according to the kth viewing angle and the second wide angle;

[0049] Before generating the fisheye image according to the position mapping relationship and the at least two images to be used, the method further includes:

[0050] Constructing a distribution characterization function according to pixel values ​​in the to-be-processed area in the kth to-be-used image, wherein the distribution characterization function is used to describe a brightness distribution state in the to-be-processed area;

[0051] For any pixel position within the to-be-processed area in the kth image to be used, the pixel value at the pixel position in the kth image to be used is updated using a weighted processing result between the pixel value at the pixel position in the kth image to be used and at least one reference pixel value corresponding to the pixel value; the weighted weight of the at least one reference pixel value is determined based on a function value corresponding to the pixel value in the distribution characterization function, and the distance between the position of the reference pixel value in the kth image to be used and the pixel position satisfies a preset distance condition.

[0052] In a possible implementation manner, for any pixel position in the to-be-processed area in the k-th image to be used, the grayscale value of the pixel position is determined according to the pixel value at the pixel position in the k-th image to be used;

[0053] The distribution characterization function is used to describe the gray value distribution state presented in the area to be processed;

[0054] The updating of the pixel value at the pixel position in the kth image to be used by using a weighted processing result between the pixel value at the pixel position in the kth image to be used and at least one reference pixel value corresponding to the pixel value comprises:

[0055] The pixel value at the pixel position in the kth image to be used is updated by utilizing the weighted processing result between the grayscale value of the pixel position in the kth image to be used and at least one reference grayscale value corresponding to the grayscale value; the at least one reference grayscale value is determined based on the at least one reference pixel value.

[0056] In a possible implementation manner, the target object is a three-dimensional house description object.

[0057] In a possible implementation manner, the at least two viewing angles include a top viewing angle, a bottom viewing angle, a front viewing angle, a back viewing angle, a left viewing angle, and a right viewing angle.

[0058] The present application provides an image generation device, comprising:

[0059] An acquisition unit, used to acquire at least two images to be used, where different images to be used are used to describe the states of the target object at different viewing angles;

[0060] A construction unit, used to construct a position mapping relationship, wherein the position mapping relationship is used to describe the correspondence between the pixel positions in the at least two images to be used and the pixel positions in the fisheye image;

[0061] A generating unit is used to generate the fisheye image according to the position mapping relationship and the at least two images to be used, wherein the pixel values ​​in the fisheye image are determined according to the pixel values ​​in the at least two images to be used, and the fisheye image is used to describe the state of the target object under at least two viewing angles.

[0062] The present application provides an electronic device, the device comprising: a processor and a memory;

[0063] The memory is used to store instructions or computer programs;

[0064] The processor is used to execute the instructions or computer programs in the memory so that the electronic device executes the image generation method provided in the present application.

[0065] The present application provides a computer-readable medium, in which instructions or computer programs are stored. When the instructions or computer programs are executed on a device, the device executes the image generation method provided by the present application.

[0066] The present application provides a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program codes for executing the image generation method provided by the present application.

[0067] Compared with the related art, this application has at least the following advantages:

[0068] In the technical solution provided by the present application, at least two images to be used are first obtained (for example, square images collected from 6 orthogonal viewing angles for the target object, etc.), so that different images to be used are used to describe the state of the target object at different viewing angles, so that the at least two images to be used can represent the state of the target object at at least two viewing angles; then a position mapping relationship is constructed, so that the position mapping relationship is used to describe the correspondence between the pixel positions in the at least two images to be used and the pixel positions in the fisheye image; finally, the fisheye image is generated based on the position mapping relationship and the at least two images to be used, so that the purpose of obtaining the fisheye image can be achieved. Among them, because the pixel values ​​in the fisheye image are determined based on the pixel values ​​in the at least two images to be used, the fisheye image can describe the state of the target object at at least two viewing angles, so that the fisheye image can better describe the target object, which is conducive to improving the effect of obtaining the fisheye image. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0070] Figure 1 A flowchart of an image generation method provided in an embodiment of the present application;

[0071] Figure 2 A schematic diagram of capturing images at different viewing angles provided in an embodiment of the present application;

[0072] Figure 3 A schematic diagram of a fisheye image provided in an embodiment of the present application;

[0073] Figure 4 A schematic diagram of an image acquisition method provided in an embodiment of the present application;

[0074] Figure 5 A schematic diagram of the structure of an image generating device provided in an embodiment of the present application;

[0075] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0076] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0077] In order to better understand the technical solution provided by the present application, the image generation method provided by the present application is described below with reference to some drawings. Figure 1 As shown, the image generation method provided in the embodiment of the present application includes the following S1-S3. Figure 1 A flowchart of an image generation method provided in an embodiment of the present application.

[0078] S1: Acquire at least two images to be used, where different images to be used are used to describe the states of the target object at different viewing angles.

[0079] The target object refers to an object that needs to be processed by fisheye image generation (for example, a three-dimensional object).

[0080] In addition, the present application does not limit the above target object. For example, in some application scenarios (such as scenarios related to three-dimensional house models, etc.), the target object can be a three-dimensional house description object (such as a virtual three-dimensional house model, a real three-dimensional house model, or a house building, etc.). Among them, the three-dimensional house description object is used to describe the state of a certain house (such as a virtual house or a real house) in a three-dimensional space, and the present application does not limit the implementation method of the three-dimensional house description object. For another example, in some application scenarios (such as scenarios related to three-dimensional building complex models, etc.), the target object can be a three-dimensional building complex description object (such as a virtual three-dimensional building complex model, a real three-dimensional building complex model, or a three-dimensional building complex, etc.). Among them, the three-dimensional building complex description object is used to describe the state of a certain building group (such as a virtual building group or a real building group, etc.) in a three-dimensional space; and the present application does not limit the three-dimensional building group description object.

[0081] Based on the above content, it can be known that in some application scenarios (such as certain design scenarios, etc.), the target object mentioned above may refer to a virtual object (such as a virtual house model, a virtual building complex model, or a virtual object model designed by relevant personnel, etc.); wherein the virtual object refers to a three-dimensional object presented in a virtual three-dimensional space. In addition, in other application scenarios (such as certain real-object photography scenarios, etc.), the target object may refer to a real object (such as a real house, a real community building complex, a real object, etc.); wherein the real object refers to a three-dimensional object presented in a real three-dimensional space.

[0082] The image to be used is obtained by using a certain image acquisition device to acquire an image of the target object mentioned above, so that the image to be used can be used as the image data required in the process of generating the fisheye image, so that the image to be used is used to provide pixel values ​​for the process of generating the fisheye image, and then the finally generated fisheye image can describe the target object in three-dimensional space (for example, Figure 2 The image acquisition device refers to a device capable of performing image acquisition processing on the target object.

[0083] It should be noted that the present application does not limit the implementation method of the above image acquisition device. For example, when the above target object is a real object, the image acquisition device may be a real camera, so that the above at least two images to be used can be acquired with the help of the real camera later. For another example, when the target object is a virtual object, the image acquisition device may be a virtual camera, so that the at least two images to be used can be acquired with the help of the virtual camera later.

[0084] In addition, for the at least two images to be used above, different images to be used are used to describe the state of the target object at different viewing angles, so that the at least two images to be used can represent the state of the target object at at least two viewing angles (for example, Figure 2 The at least two images to be used can represent the state of the target object in the three-dimensional space (for example, the top view, bottom view, front view, back view, left view and right view) as comprehensively as possible. Figure 2 The at least two viewing angles refer to the viewing angles required for image acquisition of the target object.

[0085] In addition, the present application does not limit the implementation method of at least two viewing angles in the above paragraph. For example, in some application scenarios, the at least two viewing angles above may include K viewing angles (for example, Figure 2, K is a positive integer. It can be seen that in a possible implementation, the at least two images to be used may include K images to be used, so that the at least two images to be used can describe the state of the target object at K viewing angles. The kth image to be used is used to describe the state of the target object at the kth viewing angle (for example, Figure 2 The k-th viewing angle refers to the capture viewing angle (e.g., camera viewing angle, etc.) required when capturing the k-th image to be used for the target object, and the k-th viewing angle is different from the viewing angles described by the other images to be used except the k-th image to be used in the at least two images to be used.

[0086] Based on the above content, it can be known that in some application scenarios, in order to better improve the performance of the fisheye image, the above at least two perspectives may include six orthogonal perspectives of top perspective, bottom perspective, front perspective, back perspective, left perspective and right perspective, so that the at least two images to be used acquired based on the at least two perspectives may include a top image, a bottom image, a front image, a back image, a left image and a right image, so that these images to be used (for example, Figure 2 All the square images shown in the figure can fully describe the state of the target object in the three-dimensional space, so that the fisheye image constructed based on these images to be used can represent the state of the target object in various directions in the three-dimensional space, which is conducive to improving the performance of the fisheye image. Among them, the top surface image refers to the image data collected according to the top surface perspective (for example, Figure 2 The bottom surface image refers to the image data acquired from the bottom surface perspective (for example, Figure 2 The front image refers to the image data acquired from the front view (for example, Figure 2 The back image refers to the image data acquired from the back view (for example, Figure 2 The right image refers to the image data acquired from the right viewing angle (for example, Figure 2 The left image refers to the image data acquired from the left viewing angle (for example, Figure 2 The square shown is labeled with the string "left".

[0087] It should be noted that for Figure 2For the three-dimensional space presented, the three-dimensional space may include three coordinate dimensions: X-axis, Y-axis and Z-axis. The top view is used to collect image data in the +Z-axis direction of the three-dimensional space (for example, Figure 2 The bottom view is used to collect image data in the -Z direction of the three-dimensional space (for example, Figure 2 The front view is used to collect image data in the +X axis direction of the three-dimensional space (for example, Figure 2 The back view is used to collect image data in the -X axis direction of the three-dimensional space (for example, Figure 2 The right view is used to collect image data in the +Y axis direction of the three-dimensional space (for example, Figure 2 The left viewing angle is used to collect image data in the -Y axis direction of the three-dimensional space (for example, Figure 2 The square shown is labeled with the string "left".

[0088] In addition, the present application does not limit the implementation method of the above-mentioned image to be used. For example, in some application scenarios, the image to be used may be a square image (for example, Figure 2 It should be noted that the square image refers to image data with the same width and height (for example, image data with a size of 400×400). For example, in some application scenarios, in order to ensure the fisheye image generation effect, the image to be used can be implemented using a high-definition square image.

[0089] Furthermore, the present application does not limit the size of the above-mentioned image to be used. For example, the size of the image to be used can be set in advance according to the actual application scenario.

[0090] For example, in some application scenarios, in order to achieve better flexibility while ensuring the fisheye image generation effect, the size of the above-mentioned image to be used (for example, the size of the image to be used is 4000×4000 and other constraint information) can be determined based on the pre-provided fisheye image resolution constraint (for example, the size of the fisheye image is 4000×2000 and other constraint information), so that the fisheye image constructed using these images to be used meets the fisheye image resolution constraint.

[0091] It should be noted that the present application does not limit the implementation method of the process of determining the size of the image to be used shown in the above paragraph. For example, it can be specifically: using the maximum size value (for example, 4000) appearing in the above fisheye image resolution constraint, determine the size of the image to be used, so that the size value of the image to be used in each dimension is equal to the maximum size value. For another example, it can also be specifically: searching for the square image size corresponding to the above fisheye image resolution constraint from the preset size mapping relationship as the size of the image to be used. Among them, the size mapping relationship is used to record the square image sizes corresponding to some candidate fisheye image resolution constraints; and the size mapping relationship can be constructed in advance according to the actual application scenario.

[0092] In fact, the present application does not limit the method for acquiring the at least two images to be used. For example, in some application scenarios, when the target object above is a virtual object, the process of acquiring the at least two images to be used can be specifically as follows: after rendering the target object, use a virtual camera to capture high-definition images in six directions, namely, +X axis, -X axis, +Z axis, -Z axis, +Y axis and -Y axis, as the at least two images to be used, so that these images to be used can represent the state of the target object in various positions in three-dimensional space.

[0093] It should be noted that the present application does not limit the rendering method of the virtual objects involved in the above paragraph (for example, the virtual house model involved by relevant personnel, etc.). For example, it can be implemented by any existing or future method with virtual three-dimensional model rendering function (for example, rendering with the help of a virtual engine similar to UE4, etc.).

[0094] Based on the relevant content of S1 above, it can be known that for some application scenarios (for example, the rendering construction scene of a virtual house model), after the target object (for example, a house, a building complex or an object) is rendered in a virtual three-dimensional space, a virtual camera can be used to sequentially capture high-definition images in six orthogonal directions, namely, +X axis, -X axis, +Z axis, -Z axis, +Y axis and -Y axis, as the images to be used corresponding to the target object, so that these images to be used can describe the target object at various viewing angles in the three-dimensional space (for example, Figure 2 The fisheye image of the target object can be constructed based on the images to be used.

[0095] S2: Constructing a position mapping relationship, where the position mapping relationship is used to describe the correspondence between pixel positions in at least two images to be used and pixel positions in the fisheye image.

[0096] The position mapping relationship is used to describe the corresponding relationship between the pixel positions in the at least two images to be used and the pixel positions in the fisheye image.

[0097] In addition, the present application does not limit the implementation method of the above position mapping relationship. For example, it can be implemented using any existing or future mapping method that can reflect the corresponding relationship.

[0098] In fact, in some application scenarios, in order to better improve the position mapping effect, a three-dimensional spherical surface can be used as an intermediary to realize the pixel position mapping processing between the fisheye image and the above image to be used. Based on this, the present application also provides a possible implementation of the above position mapping relationship, under which the position mapping relationship may include a first sub-mapping relationship and a second sub-mapping relationship. Among them, the first sub-mapping relationship is used to describe the correspondence between the pixel position in the fisheye image and the three-dimensional coordinate position in the three-dimensional spherical surface; the second sub-mapping relationship is used to describe the correspondence between the three-dimensional coordinate position in the three-dimensional spherical surface and the pixel position in at least two images to be used above. For ease of understanding, the relevant contents of the first sub-mapping relationship and the second sub-mapping relationship are introduced below.

[0099] The first sub-mapping relationship refers to the mapping relationship obtained by mapping each pixel position in the fisheye image to the three-dimensional coordinate system of the three-dimensional sphere, so that the first sub-mapping relationship can represent the three-dimensional coordinate position corresponding to each pixel position in the fisheye image in the three-dimensional coordinate system of the three-dimensional sphere, so that the first sub-mapping relationship can include the correspondence between each pixel position in the fisheye image and the three-dimensional coordinate position corresponding to it in the three-dimensional coordinate system of the three-dimensional sphere.

[0100] In addition, the present application does not limit the construction process of the first sub-mapping relationship above. For example, when the first sub-mapping relationship includes the correspondence between each pixel position in the fisheye image and its corresponding three-dimensional coordinate position in the three-dimensional coordinate system of the three-dimensional sphere, the construction process of the first sub-mapping relationship may specifically include the following steps 11-14.

[0101] Step 11: According to the first conversion method set in advance, the pixel position with the position index (h, v) in the above fisheye image is converted into the first spherical position representation data, and the first spherical position representation data corresponding to the "pixel position with the position index (h, v)" is obtained, so that the first spherical position representation data can express which position in the three-dimensional sphere the "pixel position with the position index (h, v)" corresponds to in the form of (azimuth, elevation, distance). Wherein, h represents the column position of the pixel position in the fisheye image, h is a positive integer, h≤H, H represents the number of pixel columns in the fisheye image (for example, the width of the fisheye image); v represents the row position of the pixel position in the fisheye image, v is a positive integer, v≤V, V represents the number of pixel rows in the fisheye image (for example, the height of the fisheye image).

[0102] The first conversion method is used to convert any pixel position in the above fisheye image into a spherical position expressed in the three-dimensional sphere in the form of (azimuth, elevation, distance).

[0103] In addition, the present application does not limit the implementation of the first conversion method mentioned above, for example, it can be set in advance according to the application scenario. For example, in some application scenarios, the first conversion method can be implemented using the following formulas (1)-(5).

[0104] R=1 (1)

[0105] Rate h =h / H (2)

[0106] Rate v =v / V (3)

[0107]

[0108] θ=Rate v ·π (5)

[0109] In the formula, Indicates the first spherical surface position representation data corresponding to the above “pixel position with the position index (h, v)”; represents the azimuth, and θ represents the elevation angle, and θ∈[0 ° ,180 °]; R represents the distance; h represents the column index identifier of the “pixel position with the position index (h, v)”, h is a positive integer, h≤H, H represents the number of pixel columns in the fisheye image (for example, the width of the fisheye image); v represents the row index identifier of the “pixel position with the position index (h, v)”, v is a positive integer, v≤V, V represents the number of pixel rows in the fisheye image (for example, the height of the fisheye image).

[0110] The first type of spherical position characterization data corresponding to the above "pixel position with the position index (h, v)" is used to indicate which position in the three-dimensional sphere the "pixel position with the position index (h, v)" corresponds to in the form of (azimuth, elevation, distance).

[0111] Based on the relevant content of step 11 above, it can be known that for any pixel position in a fisheye image (for example, a pixel position with a position index of (h, v)), the first spherical position representation data corresponding to the pixel position (for example, the position index of (h, v)) can be calculated based on the position index of the pixel position (for example, the position index of (h, v)) and the pre-set first conversion method (for example, the conversion method shown in the above formulas (1)-(5)). This data) enables the first spherical position characterization data to express in the form of (azimuth, elevation, distance) which position in the three-dimensional sphere the "pixel position with the position index (h, v)" corresponds to.

[0112] Step 12: According to the preset second conversion method, the first spherical surface position representation data corresponding to the above "pixel position with the position index (h, v)" is converted into the second spherical surface position representation data, and the second spherical surface position representation data corresponding to the "pixel position with the position index (h, v)" is obtained, so that the second spherical surface position representation data can express which position of the "pixel position with the position index (h, v)" in the three-dimensional sphere in the form of (X-axis coordinate position, Y-axis coordinate position, Z-axis coordinate position). Wherein, h is a positive integer, h≤H; v is a positive integer, v≤V.

[0113] The second conversion method is used to convert a spherical position from an expression of (azimuth, elevation, distance) to an expression of (X-axis coordinate position, Y-axis coordinate position, Z-axis coordinate position).

[0114] In addition, the present application does not limit the implementation method of the second conversion method mentioned above. For example, it can be set in advance according to the application scenario. For example, in some application scenarios, when the first spherical position representation data corresponding to the above "pixel position with the position index (h, v)" is When the data is obtained, the second conversion method can be implemented using the following formulas (6)-(8).

[0115]

[0116]

[0117] z=cosθ (8)

[0118] In the formula, (x, y, z) represents the second spherical position representation data corresponding to the above "pixel position with the position index (h, v)"; x represents the coordinate value presented on the X-axis of the three-dimensional coordinate system of the three-dimensional sphere; y represents the coordinate value presented on the Y-axis of the three-dimensional coordinate system of the three-dimensional sphere; z represents the coordinate value presented on the Z-axis of the three-dimensional coordinate system of the three-dimensional sphere.

[0119] The second type of spherical position representation data corresponding to the above "pixel position with the position index (h, v)" is used to indicate which position in the three-dimensional sphere the "pixel position with the position index (h, v)" corresponds to in the form of (X-axis coordinate position, Y-axis coordinate position, Z-axis coordinate position).

[0120] Based on the relevant content of step 12 above, it can be known that for any pixel position in a fisheye image (for example, the pixel position with the position index (h, v)), after obtaining the first spherical position representation data corresponding to the pixel position (for example, the first spherical position representation data corresponding to the "pixel position with the position index (h, v)" above), the first spherical position representation data can be converted into second spherical position representation data according to a pre-set second conversion method to obtain the second spherical position representation data corresponding to the "pixel position with the position index (h, v)", so that the second spherical position representation data can express which position in the three-dimensional sphere the "pixel position with the position index (h, v)" corresponds to in the form of (X-axis coordinate position, Y-axis coordinate position, Z-axis coordinate position), thereby achieving the purpose of mapping the two-dimensional pixel position in the fisheye image to the three-dimensional coordinate system of the three-dimensional sphere.

[0121] Step 13: Establish a corresponding relationship between the above “pixel position with the position index (h, v)” and the second spherical position representation data corresponding to the “pixel position with the position index (h, v)” as the position mapping relationship corresponding to the “pixel position with the position index (h, v)”. Wherein, h is a positive integer, h≤H; v is a positive integer, v≤V.

[0122] The position mapping relationship corresponding to the above “pixel position with the position index (h, v)” is used to indicate which position in the three-dimensional sphere the “pixel position with the position index (h, v)” corresponds to.

[0123] Based on the relevant content of step 13 above, it can be known that for any pixel position in a fisheye image (for example, a pixel position with a position index of (h, v)), after obtaining the second spherical position representation data corresponding to the pixel position (for example, the second spherical position representation data corresponding to the "pixel position with a position index of (h, v)" above), a correspondence between the pixel position and the second spherical position representation data can be created as a position mapping relationship corresponding to the pixel position, so that the position mapping relationship can indicate which position in the three-dimensional sphere the pixel position corresponds to.

[0124] Step 14: Determine a first sub-mapping relationship according to the position mapping relationship corresponding to each pixel position in the above fisheye image, so that the first sub-mapping relationship includes the position mapping relationship corresponding to each pixel position.

[0125] In the present application, for a fisheye image, after obtaining the position mapping relationships corresponding to all pixel positions in the fisheye image, these position mapping relationships can be aggregated to obtain a first sub-mapping relationship, so that the first sub-mapping relationship includes these position mapping relationships, so that the first sub-mapping relationship can indicate which position in the three-dimensional sphere each pixel position in the fisheye image corresponds to.

[0126] Based on the relevant contents of steps 11 to 14 above, it can be known that in some application scenarios, each pixel position in the fisheye image can be mapped to the three-dimensional coordinate system of the three-dimensional sphere with the help of some pre-set conversion formulas (for example, formulas (1)-(8) above, etc.) to obtain a first sub-mapping relationship, so that the first sub-mapping relationship can indicate which position in the three-dimensional coordinate system of the three-dimensional sphere each pixel position in the fisheye image corresponds to, so that the first sub-mapping relationship can be used to determine which pixel position in the at least two images to be used above each pixel position in the fisheye image corresponds to.

[0127] The second sub-mapping relationship refers to a mapping relationship obtained by mapping each spherical position in the three-dimensional spherical surface to the corresponding pixel position in the at least two images to be used above, so that the second sub-mapping relationship can represent the pixel position corresponding to each spherical position in the three-dimensional spherical surface in the at least two images to be used, so that the second sub-mapping relationship can include the correspondence between each spherical position in the three-dimensional spherical surface and the corresponding pixel position in the at least two images to be used.

[0128] In addition, the present application does not limit the construction process of the second sub-mapping relationship above. For example, when the second sub-mapping relationship includes the pixel positions corresponding to the spherical positions in the three-dimensional sphere in at least two images to be used above, the construction process of the second sub-mapping relationship can specifically include the following steps 21-23.

[0129] Step 21: For any three-dimensional coordinate position in the three-dimensional sphere above, according to the preset mapping conditions of each of the at least two images to be used above, determine the target image corresponding to the three-dimensional coordinate position from the at least two images to be used; the three-dimensional coordinate position meets the preset mapping conditions of the target image.

[0130] The preset mapping condition of the kth image to be used refers to the condition that needs to be satisfied in determining the existence of the three-dimensional spherical surface and mapped to the spherical surface position in the kth image to be used. k is a positive integer, k≤K, and K is a positive integer.

[0131] In addition, for any three-dimensional coordinate position in the three-dimensional spherical surface above, the target image corresponding to the three-dimensional coordinate position is used to indicate which image to be used the three-dimensional coordinate position is mapped to. In addition, the preset mapping condition of the target image refers to the condition that needs to be satisfied when determining the spherical position existing in the three-dimensional spherical surface and being mapped to the target image.

[0132] In addition, the present application does not limit the implementation method of the above preset mapping conditions. For ease of understanding, some examples of the above preset mapping conditions of the target image are used for explanation below.

[0133] Example 1: for any three-dimensional coordinate position in the three-dimensional sphere above, if the target image corresponding to the three-dimensional coordinate position is used to describe the state of the target object from the top surface perspective, the preset mapping condition of the target image may be: the data value in the third dimension of the three-dimensional coordinate position is greater than 0, the absolute value of the data value in the third dimension of the three-dimensional coordinate position is not less than the absolute value of the data value in the first dimension of the three-dimensional coordinate position, and the absolute value of the data value in the third dimension of the three-dimensional coordinate position is not less than the absolute value of the data value in the second dimension of the three-dimensional coordinate position.

[0134] Here, the third dimension refers to the data dimension described by the Z axis of the three-dimensional coordinate system of the three-dimensional sphere above, so that the above "data value in the third dimension" refers to the Z axis coordinate value.

[0135] The second dimension refers to the data dimension described by the Y-axis of the three-dimensional coordinate system of the three-dimensional sphere above, so that the above "data value in the second dimension" refers to the Y-axis coordinate value.

[0136] The first dimension refers to the data dimension described by the X-axis of the three-dimensional coordinate system of the three-dimensional sphere above, so that the above "data value in the first dimension" refers to the X-axis coordinate value.

[0137] Based on the relevant content of Example 1 above, it can be known that for any three-dimensional coordinate position in the three-dimensional spherical surface (for example, a spherical surface position with the three-dimensional coordinate (x, y, z)), if the three-dimensional coordinate position satisfies the condition shown in the following formula (9), it can be determined that the three-dimensional coordinate position satisfies the preset mapping condition of the top surface image, and thus it can be determined that the three-dimensional coordinate position can be mapped to the top surface image (that is, the image to be used for describing the state of the target object under the top surface perspective). The top surface image is used to describe the state of the target object under the top surface perspective. For example, the top surface image can be Figure 2 A square diagram labeled with the string "Top" is shown.

[0138] z>0,|z|≥|x|,|z|≥|y| (9)

[0139] Wherein, (x, y, z) represents the three-dimensional coordinates of a spherical position in the three-dimensional sphere; x represents the coordinate value of the spherical position on the X-axis of the three-dimensional coordinate system of the three-dimensional sphere; y represents the coordinate value of the spherical position on the Y-axis of the three-dimensional coordinate system of the three-dimensional sphere; z represents the coordinate value of the spherical position on the Z-axis of the three-dimensional coordinate system of the three-dimensional sphere; |·| represents absolute value processing.

[0140] Example 2: For any three-dimensional coordinate position in the three-dimensional sphere above, if the target image corresponding to the three-dimensional coordinate position is used to describe the state of the target object from the bottom perspective, the preset mapping condition of the target image is: the data value in the third dimension of the three-dimensional coordinate position is less than 0, the absolute value of the data value in the third dimension of the three-dimensional coordinate position is not less than the absolute value of the data value in the first dimension of the three-dimensional coordinate position, and the absolute value of the data value in the third dimension of the three-dimensional coordinate position is not less than the absolute value of the data value in the second dimension of the three-dimensional coordinate position.

[0141] It can be seen that for any three-dimensional coordinate position in the three-dimensional spherical surface (for example, a spherical surface position with the three-dimensional coordinate (x, y, z)), if the three-dimensional coordinate position satisfies the condition shown in the following formula (10), it can be determined that the three-dimensional coordinate position satisfies the preset mapping condition of the bottom image, and thus it can be determined that the three-dimensional coordinate position can be mapped to the bottom image (that is, the image to be used for describing the state of the target object under the bottom view). The bottom image is used to describe the state of the target object under the bottom view, for example, the bottom image can be Figure 2 The square diagram shown is labeled with the string "bottom".

[0142] z<0,|z|≥|x|,|z|≥|y| (10)

[0143] Wherein, (x, y, z) represents the three-dimensional coordinates of a spherical position in the three-dimensional sphere; x represents the coordinate value of the spherical position on the X-axis of the three-dimensional coordinate system of the three-dimensional sphere; y represents the coordinate value of the spherical position on the Y-axis of the three-dimensional coordinate system of the three-dimensional sphere; z represents the coordinate value of the spherical position on the Z-axis of the three-dimensional coordinate system of the three-dimensional sphere; |·| represents absolute value processing.

[0144] Example 3: For any three-dimensional coordinate position in the three-dimensional sphere above, if the target image corresponding to the three-dimensional coordinate position is used to describe the state of the target object under the frontal perspective, the preset mapping condition of the target image is: the data value in the first dimension of the three-dimensional coordinate position is greater than 0, the absolute value of the data value in the first dimension of the three-dimensional coordinate position is not less than the absolute value of the data value in the second dimension of the three-dimensional coordinate position, and the absolute value of the data value in the first dimension of the three-dimensional coordinate position is not less than the absolute value of the data value in the third dimension of the three-dimensional coordinate position.

[0145] It can be seen that for any three-dimensional coordinate position in the three-dimensional spherical surface (for example, a spherical surface position with the three-dimensional coordinates (x, y, z)), if the three-dimensional coordinate position satisfies the condition shown in the following formula (11), it can be determined that the three-dimensional coordinate position satisfies the preset mapping condition of the frontal image, and thus it can be determined that the three-dimensional coordinate position can be mapped to the frontal image (that is, the image to be used for describing the state of the target object in the frontal perspective). The frontal image is used to describe the state of the target object in the frontal perspective. For example, the frontal image can be Figure 2 The square shown is labeled with the string "front".

[0146] x>0,|x|≥|y|,|x|≥|z| (11)

[0147] Wherein, (x, y, z) represents the three-dimensional coordinates of a spherical position in the three-dimensional sphere; x represents the coordinate value of the spherical position on the X-axis of the three-dimensional coordinate system of the three-dimensional sphere; y represents the coordinate value of the spherical position on the Y-axis of the three-dimensional coordinate system of the three-dimensional sphere; z represents the coordinate value of the spherical position on the Z-axis of the three-dimensional coordinate system of the three-dimensional sphere; |·| represents absolute value processing.

[0148] Example 4: For any three-dimensional coordinate position in the three-dimensional sphere above, if the target image corresponding to the three-dimensional coordinate position is used to describe the state of the target object from a back perspective, the preset mapping condition of the target image is: the data value in the first dimension of the three-dimensional coordinate position is less than 0, the absolute value of the data value in the first dimension of the three-dimensional coordinate position is not less than the absolute value of the data value in the second dimension of the three-dimensional coordinate position, and the absolute value of the data value in the first dimension of the three-dimensional coordinate position is not less than the absolute value of the data value in the third dimension of the three-dimensional coordinate position.

[0149] It can be seen that for any three-dimensional coordinate position in the three-dimensional spherical surface (for example, a spherical surface position with the three-dimensional coordinate (x, y, z)), if the three-dimensional coordinate position satisfies the condition shown in the following formula (12), it can be determined that the three-dimensional coordinate position satisfies the preset mapping condition of the back image, and thus it can be determined that the three-dimensional coordinate position can be mapped to the back image (that is, the image to be used for describing the state of the target object at the back perspective). The back image is used to describe the state of the target object at the back perspective, and the back image can be Figure 2 The square shown is labeled with the string "Back".

[0150] x<0,|x|≥|y|,|x|≥|z| (12)

[0151] Wherein, (x, y, z) represents the three-dimensional coordinates of a spherical position in the three-dimensional sphere; x represents the coordinate value of the spherical position on the X-axis of the three-dimensional coordinate system of the three-dimensional sphere; y represents the coordinate value of the spherical position on the Y-axis of the three-dimensional coordinate system of the three-dimensional sphere; z represents the coordinate value of the spherical position on the Z-axis of the three-dimensional coordinate system of the three-dimensional sphere; |·| represents absolute value processing.

[0152] Example 5. For any three-dimensional coordinate position in the three-dimensional sphere above, if the target image corresponding to the three-dimensional coordinate position is used to describe the state of the target object in the right perspective, the preset mapping condition of the target image is: the data value in the second dimension of the three-dimensional coordinate position is greater than 0, the absolute value of the data value in the second dimension of the three-dimensional coordinate position is not less than the absolute value of the data value in the first dimension of the three-dimensional coordinate position, and the absolute value of the data value in the second dimension of the three-dimensional coordinate position is not less than the absolute value of the data value in the third dimension of the three-dimensional coordinate position.

[0153] It can be seen that for any three-dimensional coordinate position in the three-dimensional sphere (for example, a spherical position with the three-dimensional coordinate (x, y, z)), if the three-dimensional coordinate position satisfies the condition shown in the following formula (13), it can be determined that the three-dimensional coordinate position satisfies the preset mapping condition of the right image, and thus it can be determined that the three-dimensional coordinate position can be mapped to the right image (that is, the image to be used for describing the state of the target object in the right viewing angle). The right image is used to describe the state of the target object in the right viewing angle. For example, the right image can be Figure 2 The square shown is labeled with the string "right".

[0154] y>0,|y|≥|x|,|y|≥|z| (13)

[0155] Wherein, (x, y, z) represents the three-dimensional coordinates of a spherical position in the three-dimensional sphere; x represents the coordinate value of the spherical position on the X-axis of the three-dimensional coordinate system of the three-dimensional sphere; y represents the coordinate value of the spherical position on the Y-axis of the three-dimensional coordinate system of the three-dimensional sphere; z represents the coordinate value of the spherical position on the Z-axis of the three-dimensional coordinate system of the three-dimensional sphere; |·| represents absolute value processing.

[0156] Example 6. For any three-dimensional coordinate position in the three-dimensional sphere above, if the target image corresponding to the three-dimensional coordinate position is used to describe the state of the target object in the left viewing angle, the preset mapping condition of the target image is: the data value in the second dimension of the three-dimensional coordinate position is less than 0, the absolute value of the data value in the second dimension of the three-dimensional coordinate position is not less than the absolute value of the data value in the first dimension of the three-dimensional coordinate position, and the absolute value of the data value in the second dimension of the three-dimensional coordinate position is not less than the absolute value of the data value in the third dimension of the three-dimensional coordinate position.

[0157] It can be seen that for any three-dimensional coordinate position in the three-dimensional sphere (for example, a spherical position with the three-dimensional coordinate (x, y, z)), if the three-dimensional coordinate position satisfies the condition shown in the following formula (14), it can be determined that the three-dimensional coordinate position satisfies the preset mapping condition of the left image, and thus it can be determined that the three-dimensional coordinate position can be mapped to the left image (that is, the image to be used for describing the state of the target object in the left viewing angle). The left image is used to describe the state of the target object in the left viewing angle. For example, the left image can be Figure 2 The square shown is labeled with the string "Left".

[0158] y<0,|y|≥|x|,|y|≥|z| (14)

[0159] Wherein, (x, y, z) represents the three-dimensional coordinates of a spherical position in the three-dimensional sphere; x represents the coordinate value of the spherical position on the X-axis of the three-dimensional coordinate system of the three-dimensional sphere; y represents the coordinate value of the spherical position on the Y-axis of the three-dimensional coordinate system of the three-dimensional sphere; z represents the coordinate value of the spherical position on the Z-axis of the three-dimensional coordinate system of the three-dimensional sphere; |·| represents absolute value processing.

[0160] Based on the relevant contents of Examples 1 to 6 above, it can be known that for at least two images to be used above, the preset mapping conditions of different images to be used are different, so that it can be subsequently determined based on the preset mapping conditions of each image to be used whether a spherical position in the three-dimensional sphere can be mapped to the corresponding image to be used.

[0161] In addition, the present application does not limit the method for obtaining the preset mapping conditions of each image to be used. For example, in some application scenarios, the preset mapping conditions of each image to be used can be manually set in advance.

[0162] For another example, in some application scenarios, the preset mapping conditions of each image to be used can be automatically determined. It can be seen that in a possible implementation, when the above at least two images to be used include K images to be used, the process of determining the preset mapping conditions of the kth image to be used can be specifically: determining the preset mapping conditions of the kth image to be used based on the acquisition viewing angle of the kth image to be used. Among them, the acquisition viewing angle of the kth image to be used refers to the viewing angle based on which the kth image to be used is acquired, so that the kth image to be used is used to describe the state of the target object under the acquisition viewing angle. k is a positive integer, k≤K, and K is a positive integer.

[0163] It should be noted that the present application does not limit the implementation method of the step of "determining the preset mapping condition of the kth image to be used based on the acquisition viewing angle of the kth image to be used" in the above paragraph. For example, it can be specifically: if the acquisition viewing angle of the kth image to be used is a top viewing angle, then the condition shown in the above formula (9) can be determined as the preset mapping condition of the kth image to be used; if the acquisition viewing angle of the kth image to be used is a bottom viewing angle, then the condition shown in the above formula (10) can be determined as the preset mapping condition of the kth image to be used; if the acquisition viewing angle of the kth image to be used is a front viewing angle, then The condition shown in formula (11) above can be determined as the preset mapping condition for the kth image to be used; if the acquisition viewing angle of the kth image to be used is a back viewing angle, the condition shown in formula (12) above can be determined as the preset mapping condition for the kth image to be used; if the acquisition viewing angle of the kth image to be used is a right viewing angle, the condition shown in formula (13) above can be determined as the preset mapping condition for the kth image to be used; if the acquisition viewing angle of the kth image to be used is a left viewing angle, the condition shown in formula (14) above can be determined as the preset mapping condition for the kth image to be used.

[0164] Based on the relevant content of step 21 above, it can be known that in some application scenarios, when the at least two images to be used above include K images to be used, for any three-dimensional coordinate position in the three-dimensional sphere above (for example, a spherical position with a three-dimensional coordinate of (x, y, z)), it is determined whether the three-dimensional coordinate position meets the preset mapping condition of the kth image to be used. If it meets the preset mapping condition, the kth image to be used can be determined as the target image corresponding to the three-dimensional coordinate position, so that the mapping position corresponding to the three-dimensional coordinate position can be determined from the target image later. Wherein, k is a positive integer, k≤K, and K is a positive integer.

[0165] Step 22: For any three-dimensional coordinate position in the three-dimensional sphere above, determine the mapping pixel position corresponding to the three-dimensional coordinate position from the target image corresponding to the three-dimensional coordinate position.

[0166] Among them, for any three-dimensional coordinate position in the three-dimensional sphere above, the mapping pixel position corresponding to the three-dimensional coordinate position is used to indicate which pixel position in the target image corresponding to the three-dimensional coordinate position the three-dimensional coordinate position is mapped to, so that the mapping pixel position can represent the result obtained by mapping the three-dimensional coordinate position to at least two images to be used above.

[0167] In addition, the present application does not limit the implementation method of the above step 22.

[0168] In addition, in some application scenarios, in order to better improve the mapping effect, the above step 22 can be specifically as follows: for any three-dimensional coordinate position in the above three-dimensional sphere, after determining the target image corresponding to the three-dimensional coordinate position, determine the mapping pixel position corresponding to the three-dimensional coordinate position according to the preset mapping algorithm corresponding to the target image. The preset mapping algorithm refers to the algorithm required to be used when mapping a three-dimensional coordinate position to the target image.

[0169] In addition, the present application does not limit the implementation method of the above preset mapping algorithm. For ease of understanding, some examples are provided below for illustration.

[0170] Example 1: For any three-dimensional coordinate position in the three-dimensional sphere above, if the target image corresponding to the three-dimensional coordinate position is used to describe the state of the target object from the top surface perspective, the preset mapping algorithm corresponding to the target image can be specifically: first use the data value in the third dimension of the three-dimensional coordinate position to determine the data compression ratio (for example, the data compression ratio Ratio shown in formula (15) below) Top ); then, according to the data compression rate, the data value in the second dimension of the three-dimensional coordinate position, and the data value in the first dimension of the three-dimensional coordinate position, determine the mapping pixel position corresponding to the three-dimensional coordinate position in the target image (for example, the following formula (16)-(17) with (u Top , v Top )The pixel position identified by this index).

[0171] Ratio Top =1 / z (15)

[0172] u Top =(Ratio Top ×y+1) / 2 (16)

[0173] v Top =(1-Ratio Top ×x) / 2 (17)

[0174] Where (x, y, z) represents the three-dimensional coordinates of a certain spherical position in the three-dimensional sphere above; (u Top , v Top ) represents the mapping pixel position corresponding to the spherical position when the target image corresponding to the spherical position is used to describe the state of the target object under the top surface viewing angle; u Top Indicates the column index of the mapped pixel position corresponding to the spherical position in the target image; v Top Indicates the row index identifier of the mapped pixel position corresponding to the spherical position in the target image.

[0175] Based on the above two paragraphs, it can be known that in some application scenarios, for any three-dimensional coordinate position in the three-dimensional spherical surface (for example, a spherical surface position with the three-dimensional coordinates (x, y, z)), if the target image corresponding to the three-dimensional coordinate position is used to describe the state of the target object under the top surface perspective, it can be determined that the three-dimensional coordinate position can be mapped to the top surface image, so the above formulas (15)-(17) can be used to calculate the mapping pixel position corresponding to the three-dimensional coordinate position (for example, in the target image with (u Top , v Top )The pixel position identified by this index, etc.).

[0176] Example 2: For any three-dimensional coordinate position in the three-dimensional sphere above, if the target image corresponding to the three-dimensional coordinate position is used to describe the state of the target object from the bottom surface perspective, the preset mapping algorithm corresponding to the target image can be specifically: first use the data value in the third dimension of the three-dimensional coordinate position to determine the data compression ratio (for example, the data compression ratio Ratio shown in formula (18) below) Bottom ); then, according to the data compression rate, the data value in the second dimension of the three-dimensional coordinate position, and the data value in the first dimension of the three-dimensional coordinate position, determine the mapping pixel position corresponding to the three-dimensional coordinate position in the target image (for example, the following formula (19)-(20) with (u Bottom , v Bottom )The pixel position identified by this index).

[0177] Ratio Bottom = -1 / z (18)

[0178] u Bottom =(Ratio Bottom ×y+1) / 2 (19)

[0179] v Bottom =(Ratio Bottom ×x+1) / 2 (20)

[0180] Where (x, y, z) represents the three-dimensional coordinates of a certain spherical position in the three-dimensional sphere above; (u Bottom , v Bottom ) represents the mapping pixel position corresponding to the spherical position when the target image corresponding to the spherical position is used to describe the state of the target object under the bottom surface viewing angle; u Bottom Indicates the column index of the mapped pixel position corresponding to the spherical position in the target image; v Bottom Indicates the row index identifier of the mapped pixel position corresponding to the spherical position in the target image.

[0181] Based on the above two paragraphs, it can be known that in some application scenarios, for any three-dimensional coordinate position in the three-dimensional spherical surface (for example, a spherical surface position with the three-dimensional coordinates (x, y, z)), if the target image corresponding to the three-dimensional coordinate position is used to describe the state of the target object under the bottom surface perspective, it can be determined that the three-dimensional coordinate position can be mapped to the bottom surface image, so the above formulas (18)-(20) can be used to calculate the mapping pixel position corresponding to the three-dimensional coordinate position (for example, the target image with (u Bottom , v Bottom )The pixel position identified by this index, etc.).

[0182] Example 3: For any three-dimensional coordinate position in the three-dimensional sphere above, if the target image corresponding to the three-dimensional coordinate position is used to describe the state of the target object in the frontal perspective, the preset mapping algorithm corresponding to the target image can be specifically: first use the data value in the first dimension of the three-dimensional coordinate position to determine the data compression ratio (for example, the data compression ratio Ratio shown in formula (21) below) Front ); then, according to the data compression rate, the data value in the second dimension of the three-dimensional coordinate position, and the data value in the third dimension of the three-dimensional coordinate position, determine the mapping pixel position corresponding to the three-dimensional coordinate position in the target image (for example, the following formula (22)-(23) with (u Front , v Front )The pixel position identified by this index).

[0183] Ratio Front =1 / x (21)

[0184] u Front =(Ratio Front ×y+1) / 2 (22)

[0185] v Front =(Ratio Front ×z+1) / 2 (23)

[0186] Where (x, y, z) represents the three-dimensional coordinates of a certain spherical position in the three-dimensional sphere above; (u Front , v Front ) represents the mapping pixel position corresponding to the spherical position when the target image corresponding to the spherical position is used to describe the state of the target object under the frontal viewing angle; u Front Indicates the column index of the mapped pixel position corresponding to the spherical position in the target image; v Front Indicates the row index identifier of the mapped pixel position corresponding to the spherical position in the target image.

[0187] Based on the above two paragraphs, it can be known that in some application scenarios, for any three-dimensional coordinate position in the three-dimensional sphere (for example, a spherical position with the three-dimensional coordinate (x, y, z)), if the target image corresponding to the three-dimensional coordinate position is used to describe the state of the target object in the frontal perspective, it can be determined that the three-dimensional coordinate position can be mapped to the frontal image, so the above formulas (21)-(23) can be used to calculate the mapping pixel position corresponding to the three-dimensional coordinate position (for example, the target image with (u Front , v Front )The pixel position identified by this index, etc.).

[0188] Example 4: For any three-dimensional coordinate position in the three-dimensional sphere above, if the target image corresponding to the three-dimensional coordinate position is used to describe the state of the target object from the back view, the preset mapping algorithm corresponding to the target image may be: first, using the data value in the first dimension of the three-dimensional coordinate position, determine the data compression ratio (for example, the data compression ratio Ratio shown in formula (24) below): Back ); then, according to the data compression rate, the data value in the second dimension of the three-dimensional coordinate position, and the data value in the third dimension of the three-dimensional coordinate position, determine the mapping pixel position corresponding to the three-dimensional coordinate position in the target image (for example, the following formula (25)-(26) with (u Back , v Back )The pixel position identified by this index).

[0189] Ratio Back = -1 / x (24)

[0190] u Back =(1-Ratio Back ×y) / 2 (25)

[0191] v Back =(Ratio Back ×z+1) / 2 (26)

[0192] Where (x, y, z) represents the three-dimensional coordinates of a certain spherical position in the three-dimensional sphere above; (u Back , v Back ) represents the mapping pixel position corresponding to the spherical position when the target image corresponding to the spherical position is used to describe the state of the target object under the back view angle; u Back Indicates the column index of the mapped pixel position corresponding to the spherical position in the target image; v Back Indicates the row index identifier of the mapped pixel position corresponding to the spherical position in the target image.

[0193] Based on the above two paragraphs, it can be known that in some application scenarios, for any three-dimensional coordinate position in the three-dimensional spherical surface (for example, a spherical surface position with the three-dimensional coordinates (x, y, z)), if the target image corresponding to the three-dimensional coordinate position is used to describe the state of the target object in the back view, it can be determined that the three-dimensional coordinate position can be mapped to the back image, so the above formulas (24)-(26) can be used to calculate the mapping pixel position corresponding to the three-dimensional coordinate position (for example, the target image with (u Back , v Back )The pixel position identified by this index, etc.).

[0194] Example 5: For any three-dimensional coordinate position in the three-dimensional sphere above, if the target image corresponding to the three-dimensional coordinate position is used to describe the state of the target object in the right viewing angle, the preset mapping algorithm corresponding to the target image can be specifically: first use the data value in the second dimension of the three-dimensional coordinate position to determine the data compression ratio (for example, the data compression ratio Ratio shown in formula (27) below) Right ); and then determine the mapping pixel position corresponding to the three-dimensional coordinate position in the target image according to the data compression rate, the data value in the first dimension of the three-dimensional coordinate position, and the data value in the third dimension of the three-dimensional coordinate position (for example, the mapping pixel position with (u) shown in the following formulas (28)-(29) Right , v Right )The pixel position identified by this index).

[0195] Ratio Right =1 / y (27)

[0196] u Right =(1-Ratio Right ×x) / 2 (28)

[0197] v Right =(Ratio Right ×z+1) / 2 (29)

[0198] Where (x, y, z) represents the three-dimensional coordinates of a certain spherical position in the three-dimensional sphere above; (u Right , v Right ) represents the mapping pixel position corresponding to the spherical position when the target image corresponding to the spherical position is used to describe the state of the target object in the right viewing angle; u Right Indicates the column index of the mapped pixel position corresponding to the spherical position in the target image; v Right Indicates the row index identifier of the mapped pixel position corresponding to the spherical position in the target image.

[0199] Based on the above two paragraphs, it can be known that in some application scenarios, for any three-dimensional coordinate position in the three-dimensional sphere (for example, a spherical position with the three-dimensional coordinates (x, y, z)), if the target image corresponding to the three-dimensional coordinate position is used to describe the state of the target object in the right viewing angle, it can be determined that the three-dimensional coordinate position can be mapped to the right image, so the above formulas (27)-(29) can be used to calculate the mapping pixel position corresponding to the three-dimensional coordinate position (for example, the target image with (u Right , v Right )The pixel position identified by this index, etc.).

[0200] Example 6: For any three-dimensional coordinate position in the three-dimensional sphere above, if the target image corresponding to the three-dimensional coordinate position is used to describe the state of the target object in the left viewing angle, the preset mapping algorithm corresponding to the target image can be specifically: first use the data value in the second dimension of the three-dimensional coordinate position to determine the data compression ratio (for example, the data compression ratio Ratio shown in formula (30) below) Left ); and then determine the mapping pixel position corresponding to the three-dimensional coordinate position in the target image according to the data compression rate, the data value in the first dimension of the three-dimensional coordinate position, and the data value in the third dimension of the three-dimensional coordinate position (for example, the mapping pixel position with (u) shown in the following formulas (31)-(32) Left , v Left )The pixel position identified by this index).

[0201] Ratio Left =-1 / y (30)

[0202] u Left =(Ratio Left ×x+1) / 2 (31)

[0203] v Left =(Ratio Left ×z+1) / 2 (32)

[0204] Where (x, y, z) represents the three-dimensional coordinates of a certain spherical position in the three-dimensional sphere above; (u Left , v Left ) represents the mapping pixel position corresponding to the spherical position when the target image corresponding to the spherical position is used to describe the state of the target object under the left viewing angle; u Left Indicates the column index of the mapped pixel position corresponding to the spherical position in the target image; v Left Indicates the row index identifier of the mapped pixel position corresponding to the spherical position in the target image.

[0205] Based on the above two paragraphs, it can be known that in some application scenarios, for any three-dimensional coordinate position in the three-dimensional sphere (for example, a spherical position with the three-dimensional coordinates (x, y, z)), if the target image corresponding to the three-dimensional coordinate position is used to describe the state of the target object in the left viewing angle, it can be determined that the three-dimensional coordinate position can be mapped to the left image, so the above formulas (30)-(32) can be used to calculate the mapping pixel position corresponding to the three-dimensional coordinate position (for example, the target image with (u Left , v Left )The pixel position identified by this index, etc.).

[0206] Based on the relevant contents of Examples 1 to 6 above, it can be known that for the at least two images to be used above, since different images to be used are acquired at different viewing angles, so that the preset mapping algorithms corresponding to different images to be used are different, therefore, for any three-dimensional coordinate position in the three-dimensional sphere above, after obtaining the target image corresponding to the three-dimensional coordinate position, the preset mapping algorithm corresponding to the target image can be determined according to the acquisition viewing angle of the target image, so that the mapping pixel position corresponding to the three-dimensional coordinate position in the target image can be calculated based on the preset mapping algorithm corresponding to the target image.

[0207] It should be noted that the present application does not limit the implementation method of the step of "determining the preset mapping algorithm corresponding to the target image according to the acquisition viewing angle of the target image" in the above paragraph. For example, it can be specifically: if the acquisition viewing angle of the target image is a top viewing angle, then the above formulas (15)-(17) can be determined as the preset mapping algorithm corresponding to the target image; if the acquisition viewing angle of the target image is a bottom viewing angle, then the above formulas (18)-(20) can be determined as the preset mapping algorithm corresponding to the target image; if the acquisition viewing angle of the target image is a front viewing angle, then The above formulas (21)-(23) can be determined as the preset mapping algorithm corresponding to the target image; if the acquisition viewing angle of the target image is a back viewing angle, the above formulas (24)-(26) can be determined as the preset mapping algorithm corresponding to the target image; if the acquisition viewing angle of the target image is a right viewing angle, the above formulas (27)-(29) can be determined as the preset mapping algorithm corresponding to the target image; if the acquisition viewing angle of the target image is a left viewing angle, the above formulas (30)-(32) can be determined as the preset mapping algorithm corresponding to the target image.

[0208] Based on the relevant content of the preset mapping algorithm above, it can be known that in some application scenarios, for any three-dimensional coordinate position in the three-dimensional sphere above, after determining the target image corresponding to the three-dimensional coordinate position, the mapping pixel position corresponding to the three-dimensional coordinate position can be determined from the target image according to the preset mapping algorithm corresponding to the target image; and the mapping pixel position determination process corresponding to the three-dimensional coordinate position can specifically include the following steps 221-222.

[0209] Step 221: For any three-dimensional coordinate position in the three-dimensional sphere above, determine the data compression rate corresponding to the three-dimensional coordinate position based on the data value in the target dimension in the three-dimensional coordinate position; the target dimension is determined based on the acquisition viewing angle of the target image corresponding to the three-dimensional coordinate position.

[0210] The target dimension refers to the data dimension that needs to participate in the compression ratio calculation process when mapping a three-dimensional coordinate position to a target image corresponding to the three-dimensional coordinate position.

[0211] In addition, for the target dimension corresponding to any three-dimensional coordinate position in the three-dimensional sphere above, the target dimension can be determined based on the acquisition viewing angle of the target image corresponding to the three-dimensional coordinate position; and the determination process can be specifically: if the acquisition viewing angle of the target image is a top viewing angle or a bottom viewing angle, then the target dimension can be the third dimension (for example, the data dimension represented by the Z axis); if the acquisition viewing angle of the target image is a front viewing angle or a back viewing angle, then the target dimension can be the first dimension (for example, the data dimension represented by the X axis); if the acquisition viewing angle of the target image is a right viewing angle or a left viewing angle, then the target dimension can be the second dimension (for example, the data dimension represented by the Y axis).

[0212] The data compression rate refers to the compression ratio required to map a three-dimensional coordinate position to the target image corresponding to the three-dimensional coordinate position; and the present application does not limit the determination process of the data compression rate. For example, it can be specifically: calculating the ratio between 1 and the data value in the target dimension of the three-dimensional coordinate position (that is, 1 / the data value in the target dimension of the three-dimensional coordinate position); and then determining the absolute value of the ratio as the data compression rate corresponding to the three-dimensional coordinate position. It can be seen that in one possible implementation, the data compression rate = 1 / |the data value in the target dimension of the three-dimensional coordinate position|.

[0213] Based on the above content, it can be known that in some application scenarios, for any three-dimensional coordinate position in the three-dimensional spherical surface mentioned above, after determining the target image corresponding to the three-dimensional coordinate position, if the target image is used to describe the state of the target object in the top view, the data compression rate corresponding to the three-dimensional coordinate position can be determined by means of the above formula (15); if the target image is used to describe the state of the target object in the bottom view, the data compression rate corresponding to the three-dimensional coordinate position can be determined by means of the above formula (18); if the target image is used to describe the state of the target object in the front view, the three The data compression rate corresponding to the three-dimensional coordinate position can be determined with the help of the above formula (21); if the target image is used to describe the state of the target object in the back perspective, the data compression rate corresponding to the three-dimensional coordinate position can be determined with the help of the above formula (24); if the target image is used to describe the state of the target object in the right perspective, the data compression rate corresponding to the three-dimensional coordinate position can be determined with the help of the above formula (27); if the target image is used to describe the state of the target object in the left perspective, the data compression rate corresponding to the three-dimensional coordinate position can be determined with the help of the above formula (30).

[0214] Based on the relevant content of step 221 above, it can be known that for any three-dimensional coordinate position in the three-dimensional sphere above, after obtaining the target image corresponding to the three-dimensional coordinate position, the target dimension corresponding to the three-dimensional coordinate position can be determined based on the acquisition viewing angle of the target image; and then the data compression rate corresponding to the three-dimensional coordinate position can be determined based on the data value of the target dimension in the three-dimensional coordinate position, so that the mapping pixel position corresponding to the three-dimensional coordinate position in the target image can be determined based on the data compression rate.

[0215] Step 222: For any three-dimensional coordinate position in the above three-dimensional sphere, determine the mapping pixel position corresponding to the three-dimensional coordinate position based on the data compression rate corresponding to the three-dimensional coordinate position and the other two data values ​​in the three-dimensional coordinate position except the data value in the target dimension.

[0216] It should be noted that the present application does not limit the implementation method of the above step 222. For example, if the above data compression rate is determined by the above formula (15), then the step 222 can be implemented by the above formulas (16)-(17); if the above data compression rate is determined by the above formula (18), then the step 222 can be implemented by the above formulas (19)-(20); if the above data compression rate is determined by the above formula (21), then the step 222 can be implemented by the above formula (22). The step 222 can be implemented by using formulas (22)-(23); if the data compression ratio is determined by using formula (24), the step 222 can be implemented by using formulas (25)-(26); if the data compression ratio is determined by using formula (27), the step 222 can be implemented by using formulas (28)-(29); if the data compression ratio is determined by using formula (30), the step 222 can be implemented by using formulas (31)-(32).

[0217] Based on the relevant contents of steps 221 to 222 above, it can be known that for any three-dimensional coordinate position in the three-dimensional sphere above, after obtaining the target image corresponding to the three-dimensional coordinate position, the target dimension corresponding to the three-dimensional coordinate position can be determined based on the acquisition viewing angle of the target image; then, based on the data value in the target dimension in the three-dimensional coordinate position, the data compression rate corresponding to the three-dimensional coordinate position can be determined; then, based on the data compression rate and the other two data values ​​in the three-dimensional coordinate position except the data value in the target dimension, the mapping pixel position corresponding to the three-dimensional coordinate position can be determined.

[0218] In fact, in some application scenarios (for example, scenarios where the resolution of the image to be used is relatively low, etc.), in order to better avoid the aliasing problem, the present application also provides a possible implementation of the above step 22. In this implementation, the step 22 can specifically include the following steps 223-226.

[0219] Step 223: For any three-dimensional coordinate position in the three-dimensional sphere above, determine the data compression rate corresponding to the three-dimensional coordinate position based on the data value in the target dimension in the three-dimensional coordinate position; the target dimension is determined based on the acquisition viewing angle of the target image corresponding to the three-dimensional coordinate position.

[0220] It should be noted that the relevant content of step 223 can be found in the above step 221, and for the sake of brevity, it will not be repeated here.

[0221] Step 224: For any three-dimensional coordinate position in the above three-dimensional sphere, determine the theoretical mapping position corresponding to the three-dimensional coordinate position based on the data compression rate corresponding to the three-dimensional coordinate position and the other two data values ​​in the three-dimensional coordinate position except the data value in the target dimension.

[0222] The “theoretical mapping position corresponding to the three-dimensional coordinate position” mentioned above refers to the mapping position directly calculated using the formula.

[0223] In addition, the implementation method of the acquisition process of the above “theoretical mapping position corresponding to the three-dimensional coordinate position” is similar to the implementation method of the above step 222, and for the sake of brevity, it will not be repeated here.

[0224] Step 225: For any three-dimensional coordinate position in the three-dimensional sphere above, based on the theoretical mapping position corresponding to the three-dimensional coordinate position, search for at least one adjacent pixel position corresponding to the theoretical mapping position from the target image corresponding to the three-dimensional coordinate position as the at least one adjacent pixel position corresponding to the three-dimensional coordinate position.

[0225] The adjacent pixel position refers to the pixel position that actually exists in the target image and is close to the theoretical mapping position.

[0226] In addition, the present application does not limit the number of adjacent pixel positions in the above "at least one adjacent pixel position". For example, in some application scenarios, in order to take into account both performance and efficiency, the above "at least one adjacent pixel position" may include 4 adjacent pixel positions. It can be seen that in a possible implementation, for any three-dimensional coordinate position in the above three-dimensional sphere, if the theoretical mapping position corresponding to the three-dimensional coordinate position is (first non-integer, second non-integer), then the at least one adjacent pixel position corresponding to the theoretical mapping position may include (first integer, second integer), (first integer, third integer), (fourth integer, second integer) and (fourth integer, third integer). Among them, the first integer refers to the integer closest to the first non-integer and less than the first non-integer; the fourth integer refers to the integer closest to the first non-integer and greater than the first non-integer; the second integer refers to the integer closest to the second non-integer and less than the second non-integer; the third integer refers to the integer closest to the second non-integer and greater than the second non-integer.

[0227] In addition, the present application does not limit the determination process of the above “at least one adjacent pixel position”. For example, it may be specifically implemented by adopting some existing or future adjacent pixel determination methods.

[0228] Based on the relevant content of step 225 above, it can be known that for any three-dimensional coordinate position in the three-dimensional sphere above, after calculating the theoretical mapping position corresponding to the three-dimensional coordinate position using the target image corresponding to the three-dimensional coordinate position and some formulas corresponding to the target image (for example, formulas (15)-(17) above, formulas (18)-(20) above, formulas (21)-(23) above, formulas (24)-(26) above, formulas (27)-(29) above, or formulas (30)-(32) above, etc.), at least one adjacent pixel position corresponding to the theoretical mapping position can be found from the target image, so that the mapping pixel position corresponding to the three-dimensional coordinate position in the target image can be determined based on these adjacent pixel positions.

[0229] Step 226: For any three-dimensional coordinate position in the three-dimensional sphere, determine a mapping pixel position corresponding to the three-dimensional coordinate position according to at least one adjacent pixel position corresponding to the three-dimensional coordinate position.

[0230] It should be noted that the present application does not limit the implementation method of the above step 226. For example, it can be specifically as follows: for any three-dimensional coordinate position in the above three-dimensional sphere, after obtaining at least one adjacent pixel position corresponding to the three-dimensional coordinate position, these adjacent pixel positions are aggregated to obtain the mapped pixel position corresponding to the three-dimensional coordinate position, so that the mapped pixel position includes these adjacent pixel positions.

[0231] Based on the relevant contents of steps 223 to 226 above, it can be known that in some application scenarios, for any three-dimensional coordinate position in the three-dimensional spherical surface above, after determining the target image corresponding to the three-dimensional coordinate position, the theoretical mapping position corresponding to the three-dimensional coordinate position can be calculated according to some formulas corresponding to the target image (for example, formulas (15)-(17), (18)-(20), (21)-(23), (24)-(26), (27)-(29), or (30)-(32)); then, at least one adjacent pixel position corresponding to the theoretical mapping position is searched from the target image; finally, based on the at least one adjacent pixel position, the mapping pixel position corresponding to the three-dimensional coordinate position is determined so that the mapping pixel position includes the at least one adjacent pixel position. In this way, the aliasing problem caused by the phenomenon that multiple spherical positions in the three-dimensional spherical surface correspond to the same position in the target image when the resolution of the target image is low can be effectively avoided, thereby facilitating the improvement of the fisheye image generation effect.

[0232] Based on the relevant content of step 22 above, it can be known that in some application scenarios, for any three-dimensional coordinate position in the three-dimensional sphere above, after obtaining the target image corresponding to the three-dimensional coordinate position, the mapping pixel position corresponding to the three-dimensional coordinate position can be determined from the target image, so that the mapping pixel position can indicate which pixel position in the target image the three-dimensional coordinate position is mapped to.

[0233] Step 23: For any three-dimensional coordinate position in the above three-dimensional sphere, determine a second sub-mapping relationship based on the correspondence between the three-dimensional coordinate position and the mapping pixel position corresponding to the three-dimensional coordinate position, so that the second sub-mapping relationship includes the correspondence between the three-dimensional coordinate position and the mapping pixel position corresponding to the three-dimensional coordinate position.

[0234] In the present application, for the above-mentioned three-dimensional spherical surface, after obtaining the mapping pixel positions corresponding to each three-dimensional coordinate position in the three-dimensional spherical surface, a second sub-mapping relationship can be constructed based on the correspondence between these three-dimensional coordinate positions and their corresponding mapping pixel positions, so that the second sub-mapping relationship includes these correspondences, thereby enabling the second sub-mapping relationship to represent the pixel positions corresponding to each spherical position in the three-dimensional spherical surface in the at least two images to be used.

[0235] Based on the relevant contents of steps 21 to 23 above, it can be known that for the three-dimensional sphere above, the mapping process shown in steps 21 to 23 can be used to map each spherical position in the three-dimensional sphere to the corresponding pixel position in at least two images to be used above to obtain a second sub-mapping relationship, so that the second sub-mapping relationship includes these corresponding relationships, so that the second sub-mapping relationship can represent the pixel positions corresponding to each spherical position in the three-dimensional sphere in the at least two images to be used.

[0236] Based on the relevant contents of the first sub-mapping relationship and the second sub-mapping relationship above, it can be known that in one possible implementation, the position mapping relationship above may include the first sub-mapping relationship and the second sub-mapping relationship, so that the position mapping relationship can express the correspondence between the pixel positions in at least two images to be used and the pixel positions in the fisheye image by means of two mapping methods.

[0237] Based on the relevant content of S2 above, it can be known that in some application scenarios, if you want to generate a fisheye image for a target object, then after obtaining at least two images to be used that are collected for the target object, you can construct a position mapping relationship based on these images to be used and the resolution requirements of the fisheye image, so that the position mapping relationship can represent the correspondence between the pixel positions in these images to be used and the pixel positions in the fisheye image, so that the pixel values ​​at each pixel position in the fisheye image can be determined based on the position mapping relationship and the pixel values ​​in these images to be used.

[0238] S3: Generate a fisheye image based on the position mapping relationship and at least two images to be used, wherein the pixel values ​​in the fisheye image are determined based on the pixel values ​​in the at least two images to be used, and the fisheye image is used to describe the state of the target object under at least two viewing angles.

[0239] The fisheye image is used to describe the state of the target object under at least two viewing angles. For example, when the at least two images to be used include Figure 2 When the 6 square diagrams are shown, the fisheye diagram can be Figure 3 The fisheye image shown can describe the state of the target object in various directions.

[0240] In addition, the present application does not limit the generation process of the above fisheye image. For example, it can be specifically as follows: for any pixel position in the fisheye image, first determine the pixel position corresponding to the pixel position in the at least two images to be used above based on the above position mapping relationship; then determine the pixel value that needs to be filled in the pixel position in the fisheye image based on the pixel value at the "corresponding pixel position" in the corresponding image to be used, so that the purpose of generating a fisheye image can be achieved.

[0241] In addition, the present application does not limit the process of determining the pixel value to be filled at any pixel position in the above fisheye image (for example, a pixel position with a position index of (h, v), etc.). For example, if the above position mapping relationship indicates that the pixel position only corresponds to a pixel position in a certain image to be used (for example, Figure 2 The bottom image shown has (u Bottom , v Bottom ) the pixel position identified by this index, etc.), then the corresponding pixel position in the image to be used (for example, the pixel position with (u Bottom , v Bottom ) can be directly determined as the pixel value that needs to be filled at this pixel position in the fisheye image.

[0242] For another example, for any pixel position in the above fisheye image (e.g., a pixel position with a position index of (h, v), etc.), if the above position mapping relationship indicates that the pixel position corresponds to multiple pixel positions in a certain image to be used (e.g., "at least one adjacent pixel position" above), the pixel values ​​at the multiple pixel positions can be weighted summed to obtain the pixel value to be filled at the pixel position in the fisheye image. For ease of understanding, the following is an explanation with examples.

[0243] As an example, for any pixel position in the above fisheye image (for example, a pixel position with a position index of (h, v)), if the above position mapping relationship indicates that the pixel position corresponds to a three-dimensional coordinate position in a three-dimensional sphere (for example, a spherical position with a coordinate of (x, y, z)), and the mapped pixel position corresponding to the three-dimensional coordinate position in its corresponding target image includes at least one adjacent pixel position above, then the pixel value at the pixel position corresponding to the three-dimensional coordinate position in the fisheye image (for example, the pixel position with a position index of (h, v)) can be obtained by weighted summing the pixel values ​​at the at least one adjacent pixel position in the target image; and for any adjacent pixel position, the weighted weight corresponding to the pixel value at the adjacent pixel position in the target image is determined based on the distance between the adjacent pixel position and the theoretical mapping position corresponding to the three-dimensional coordinate position above. It can be seen that in a possible implementation, the pixel value required to be filled at any pixel position in the fisheye image can be determined using the following formulas (33)-(34).

[0244]

[0245]

[0246] Where (h, v) represents a pixel position in the fisheye image; PixelValue(h, v) represents the pixel value to be filled at the pixel position in the fisheye image; {pixelpos1, pixelpos2, ..., pixelpos T} represents the mapping pixel position corresponding to the pixel position in a certain image to be used (for example, the target image above) (that is, at least one adjacent pixel position above); pixelpos t Indicates the tth adjacent pixel position corresponding to the pixel position in the image to be used (for example, the target image above), where t is a positive integer, t≤T; PixelValue(pixelpos t ) indicates that the image to be used is at pixelpos t The pixel value on thet ) indicates that the PixelValue(pixelpos t ) corresponding weighted weight; pixelpos theroy represents the theoretical mapping position corresponding to the three-dimensional coordinate position of the pixel position in the three-dimensional sphere and in the image to be used; distance(pixelpos t ,pixelpos theroy ) indicates that the pixelpos t With the pixelpos theroy The distance between pixels f represents the fth adjacent pixel position corresponding to the pixel position in the image to be used (for example, the target image above), where f is a positive integer and f≤T; distance(pixelpos f ,pixelpos theroy ) indicates that the pixelpos f With the pixelpos theroy The distance between.

[0247] Based on the relevant content of the fisheye image above, it can be known that in some application scenarios, if you want to generate a fisheye image for a target object, then after obtaining at least two images to be used collected for the target object, you can first construct a position mapping relationship based on the images to be used and the resolution of the fisheye image, so that the position mapping relationship can represent the correspondence between the pixel positions in the images to be used and the pixel positions in the fisheye image; then, based on the position mapping relationship and the pixel values ​​in the images to be used, determine the pixel values ​​to be filled in each pixel position in the fisheye image; finally, fill the pixel values ​​to be filled in each pixel position in the fisheye image that are finally determined to be filled in the corresponding pixel positions in the fisheye image, and obtain a fisheye image (for example, Figure 3 Thus, the purpose of constructing a fisheye image using some images can be achieved.

[0248] Based on the relevant contents of S1 to S3 above, it can be known that for the image generation method provided in the embodiment of the present application, at least two images to be used are first obtained (for example, square images collected from 6 orthogonal viewing angles for the target object, etc.), so that different images to be used are used to describe the state of the target object at different viewing angles, so that the at least two images to be used can represent the state of the target object at at least two viewing angles; then a position mapping relationship is constructed, so that the position mapping relationship is used to describe the correspondence between the pixel positions in the at least two images to be used and the pixel positions in the fisheye image; finally, based on the position mapping relationship and the at least two images to be used, the fisheye image is generated, so that the purpose of obtaining the fisheye image can be achieved. Among them, because the pixel values ​​in the fisheye image are determined based on the pixel values ​​in the at least two images to be used, the fisheye image can describe the state of the target object at at least two viewing angles, so that the fisheye image can better describe the target object, which is conducive to improving the effect of obtaining the fisheye image.

[0249] Research has found that in some application scenarios, the central area of ​​the viewport of the image acquisition device (for example, a virtual camera, etc.) is rendered brighter while the edge is rendered darker. As a result, the central image area of ​​the image data (for example, the image to be used above) acquired by the image acquisition device is brighter but the edge image area is darker. As a result, the fisheye image constructed based on the acquired image data may show obvious dark lines, which leads to stitching problems.

[0250] In order to solve the stitching problem described in the previous paragraph, the present application provides a possible implementation of the above image generation method. Under this implementation, when the construction requirement of the above fisheye image includes a pre-set second wide angle, and the second wide angle is used to describe the area range of the pixel values ​​required to be referenced when constructing the fisheye image, the image generation method may include the following steps 31-35.

[0251] Step 31: Acquire K images to be used, wherein the kth image to be used is obtained by capturing the target object according to the kth viewing angle and the first wide angle, the second wide angle is smaller than the first wide angle, k is a positive integer, k≤K.

[0252] Among them, the second wide angle is determined based on the pixel value collection requirements required when constructing the fisheye image, so that the second wide angle can describe the collection area range of the pixel values ​​required to be referenced when constructing the fisheye image, so that the second wide angle can represent the minimum wide angle required when collecting image data required for the fisheye image construction process; and the present application does not limit the second wide angle. For example, in some application scenarios, the second wide angle can be 90 degrees.

[0253] The first wide angle refers to the wide angle required to be used in the image acquisition process when overcoming the stitching problem by acquiring multiple pixel values, so that the pixel values ​​in the image to be used acquired according to the first wide angle are greater than the pixel values ​​required to be referenced when constructing the fisheye image, so that the pixel values ​​required to be referenced when constructing the fisheye image will not appear in the edge area of ​​the image to be used.

[0254] In addition, the present application does not limit the implementation method of the first wide angle above. For example, in some application scenarios, if the fisheye image construction process needs to refer to the pixel values ​​in some square images with a wide angle of 90 degrees, in order to overcome the above stitching problem, the first wide angle can be 120 degrees, so that the image data collected according to the first wide angle (for example, by Figure 4 The image data shown in the middle area 401) can not only meet the pixel value requirements in the fisheye image construction process (for example, the pixel value requirements within the wide angle range of 90 degrees), but also ensure that the pixel values ​​that meet the pixel value requirements in the image data obtained by the first wide angle acquisition are all located in the non-edge area of ​​the image area (for example, Figure 4 area 402 as shown).

[0255] Based on the relevant content of step 31 above, it can be known that for some application scenarios, if it is necessary to perform image acquisition processing at K viewing angles for the target object, the image acquisition processing can be performed on the target object according to the kth viewing angle and the first wide angle to obtain the kth image to be used, so that the kth image to be used can be the state of the target object at the kth viewing angle, and the kth image to be used can be recorded in the non-edge area to meet the pixel value requirements in the fisheye image construction process, so that the stitching problem can be effectively avoided. Wherein, k is a positive integer, k≤K.

[0256] Step 32: construct a position mapping relationship, which is used to describe the correspondence between the pixel positions in the above K images to be used and the pixel positions in the fisheye image.

[0257] It should be noted that the implementation of step 32 is similar to the implementation of S2 above, and for the sake of brevity, it will not be repeated here.

[0258] Step 33: According to the image size of the kth image to be used, determine the image region description information corresponding to the second wide angle in the kth image to be used, where k is a positive integer, k≤K.

[0259] The image size of the kth image to be used is used to describe the size of the kth image to be used.

[0260] The above “image region description information corresponding to the second wide angle in the k-th image to be used” is used to describe the image region existing in the k-th image to be used and meeting the pixel value requirements for constructing the fisheye image, so that the “image region description information corresponding to the second wide angle in the k-th image to be used” is used to describe the non-edge region in the k-th image to be used (for example, Figure 4 area 402 as shown).

[0261] In addition, the present application does not limit the above “image region description information corresponding to the second wide angle in the kth image to be used”, for example, it may include the size of the discarded region and / or the size of the retained region. The following introduces these two data respectively.

[0262] The discarded area size is used to indicate the size of the image area that does not need to be retained in the kth image to be used (that is, the image area that does not meet the pixel value requirements when constructing the fisheye image); and the present application does not limit the discarded area size, for example, the discarded area size may include a first width (for example, Figure 4 W1 shown) and a first height (e.g., Figure 4 H1 shown), the first width refers to the image area that does not need to be retained in the k-th image to be used (for example, Figure 4 The first height refers to half of the width of the other areas in the area 401 except the area 402, and the first height refers to half of the height of the image area that does not need to be retained in the kth image to be used.

[0263] In addition, the present application does not limit the method for determining the size of the discarded area described above. For example, it can be implemented using the following formulas (35)-(36).

[0264] W1=Width original / 2-Width original / (2×tan(60)) (35)

[0265] H1=Height original / 2-Height original / (2×tan(60)) (36)

[0266] Where W1 represents the first width of the above text; H1 represents the first height of the above text; Width original Indicates the width of the kth image to be used above; Height original Indicates the height of the kth image to be used above.

[0267] The reserved area size is used to indicate the size of the image area that needs to be reserved for use in the kth image to be used (that is, the image area that meets the pixel value requirements when constructing the fisheye image); and the present application does not limit the reserved area size, for example, it may specifically include the second width (for example, Figure 4 W2 shown) and a second height (e.g., Figure 4 As shown in H2), the second width refers to the width of the image area that needs to be retained for use in the k-th image to be used, and the second height refers to the height of the image area that needs to be retained for use in the k-th image to be used.

[0268] In addition, the present application does not limit the method for determining the size of the reserved area mentioned above. For example, it can be implemented using the following formulas (37)-(38).

[0269] W2=Width original -2×W1 (37)

[0270] H2=Height original -2×H1 (38)

[0271] Where W2 represents the second width of the above text; H2 represents the second height of the above text; W1 represents the first width of the above text; H1 represents the first height of the above text; Width original Indicates the width of the kth image to be used above; Height original Indicates the height of the kth image to be used above.

[0272] Based on the relevant contents of step 33 above, it can be known that for the kth image to be used above, since the kth image to be used is collected according to the first wide angle, so that the pixel value recorded in the kth image to be used is more than the pixel value required for reference when the fisheye image is constructed, after the kth image to be used is obtained, the image area description information corresponding to the second wide angle in the kth image to be used is determined according to the image size of the kth image to be used (for example, Figure 4 W1, W2, H1, H2, etc. shown in the figure) so that the pixel values ​​required for reference when constructing the fisheye image can be extracted from the kth image to be used according to the image area description information. Wherein, k is a positive integer, k≤K.

[0273] Step 34: extract pixel values ​​from the kth image to be used according to the image region description information corresponding to the second wide angle in the kth image to be used, and obtain pixel values ​​corresponding to the second wide angle in the kth image to be used, where k is a positive integer, k≤K.

[0274] The pixel value corresponding to the second wide angle in the kth image to be used refers to the pixel value existing in the kth image to be used and located within the second wide angle range, so that the "pixel value corresponding to the second wide angle in the kth image to be used" can represent the pixel value existing in the kth image to be used and meets the requirements of fisheye image construction. For example, when the kth image to be used is Figure 4 When the square diagram is shown, the "pixel value corresponding to the second wide angle in the kth image to be used" may refer to Figure 4 The pixel values ​​within the middle area 402.

[0275] In addition, the present application does not limit the implementation method of the above step 34. For example, in some application scenarios, the step 34 may specifically be: after obtaining the image area description information corresponding to the second wide angle in the above k-th image to be used, the k-th image to be used may be cropped according to the image area description information to obtain a reserved area image, so that the reserved area image can represent the image area that needs to be reserved for use in the k-th image to be used (for example, Figure 4 Then, the pixel value in the reserved area image is determined as the pixel value corresponding to the second wide angle in the k-th image to be used.

[0276] For another example, in some application scenarios, in order to save resources (such as storage resources, computing resources, etc.), pixel values ​​that meet the requirements of fisheye image construction can be directly obtained from the kth image to be used above. Based on this, it can be seen that in a possible implementation, the above step 34 can be specifically: after obtaining the image area description information corresponding to the second wide angle in the kth image to be used above, the pixel value that meets the image area description information can be searched from the kth image to be used according to the image area description information as the pixel value corresponding to the second wide angle in the kth image to be used, which can effectively avoid the resource consumption defect caused by cropping the reserved area image from the kth image to be used, thereby helping to reduce resource consumption.

[0277] Based on the relevant contents of step 34 above, it can be known that for the kth image to be used above, since the kth image to be used is collected according to the first wide angle, so that the pixel values ​​recorded in the kth image to be used are more than the pixel values ​​required for reference when constructing the fisheye image, after obtaining the kth image to be used and the image area description information corresponding to the second wide angle in the kth image to be used, the pixel values ​​corresponding to the second wide angle in the kth image to be used can be extracted from the kth image to be used according to the image area description information, so that the "pixel values ​​corresponding to the second wide angle in the kth image to be used" can represent the pixel values ​​existing in the kth image to be used and meet the requirements of fisheye image construction, so that the "pixel values ​​corresponding to the second wide angle in the kth image to be used" can be used to determine the pixel values ​​required to be filled in certain pixel positions in the fisheye image. Wherein, k is a positive integer, k≤K.

[0278] Step 35: Generate a fisheye image based on the above position mapping relationship and the pixel values ​​corresponding to the second wide angle in the K images to be used.

[0279] In the present application, for the above K images to be used, after extracting the pixel values ​​corresponding to the second wide angle in each image to be used, the pixel values ​​required to be filled in each pixel position in the fisheye image can be determined based on these pixel values ​​and the above position mapping relationship; and then the determined pixel values ​​are filled into the corresponding pixel positions in the fisheye image to obtain the fisheye image.

[0280] Based on the relevant contents of steps 31 to 35 above, it can be known that in some application scenarios, in order to better overcome the above-mentioned stitching problem, excessive pixel value acquisition can be performed in the image acquisition stage (for example, if the fisheye image construction process needs to refer to the pixel values ​​within a wide-angle range of 90 degrees, the image acquisition processing can be performed according to a wide-angle range of 120 degrees, so that the pixel values ​​in the collected image data are more than the pixel values ​​within the wide-angle range of 90 degrees, etc.), so that the collected pixel values ​​become more, so that the pixel values ​​required for reference when constructing the fisheye image are all presented in the non-edge area of ​​the collected image data, and then, when the fisheye image is subsequently constructed, only the pixel values ​​in the non-edge area of ​​the image data are used, and there is no need to use the pixel values ​​in the edge area of ​​the image data, thereby effectively avoiding the stitching problem caused by the use of the pixel values ​​in the edge area of ​​the image data.

[0281] In fact, in some application scenarios, the above-mentioned stitching problem can be overcome by fading the pixel values ​​in the edge area of ​​the image data. Based on this, the present application also provides a possible implementation of the above-mentioned image generation method. In this implementation, when the construction requirements of the above-mentioned fisheye image include a pre-set second wide angle, and the second wide angle is used to describe the area range of the pixel values ​​required to be referenced when constructing the fisheye image, the image generation method can include the following steps 41 to 44.

[0282] Step 41: Acquire K images to be used, wherein the kth image to be used is obtained by capturing the target object according to the kth viewing angle and the second wide angle, k is a positive integer, k≤K.

[0283] In the present application, for some application scenarios, if it is necessary to perform image acquisition processing under K viewing angles for the target object, the image acquisition processing can be performed on the target object according to the kth viewing angle and the second wide angle to obtain the kth image to be used, so that the kth image to be used can be the state of the target object at the kth viewing angle, and the kth image to be used can record all pixel values ​​within the second wide angle range, so that the kth image to be used can reduce resource consumption (such as storage resources, etc.) as much as possible under the premise of meeting the construction requirements of the above fisheye image. Wherein, k is a positive integer, k≤K.

[0284] Step 42: construct a position mapping relationship, which is used to describe the correspondence between the pixel positions in the above K images to be used and the pixel positions in the fisheye image.

[0285] It should be noted that the implementation of step 42 is similar to the implementation of S2 above, and for the sake of brevity, it will not be repeated here.

[0286] Step 43: Based on the pixel values ​​in the to-be-processed area in the kth to-be-used image, a distribution characterization function corresponding to the kth to-be-used image is constructed, and the distribution characterization function is used to describe the brightness distribution state in the to-be-processed area.

[0287] The area to be processed refers to the area that exists in the k-th image to be used above and needs to be faded; and the present application does not limit the area to be processed. For example, the area to be processed may include the edge area in the k-th image to be used. It should be noted that the present application does not limit the method for obtaining the edge area. For example, the edge area can be determined based on a preset edge area size. For another example, the process of obtaining the edge area can also be: automatically analyzing the dark area of ​​the pixel values ​​in the image data to obtain the edge area.

[0288] The distribution characterization function is used to represent the brightness distribution state in the area to be processed in the kth image to be used above; and the present application does not limit the implementation method of the distribution characterization function, for example, it can be implemented with the help of a Gaussian filter function.

[0289] In fact, in some application scenarios, the brightness of image data can be characterized by grayscale values. Therefore, in order to achieve fading processing while retaining the pixel color as much as possible, the grayscale value fading method (for example, the grayscale value fading process shown in formulas (39)-(40) below) can be used to complete the update processing of the pixel values ​​in the to-be-processed area in the kth image to be used above. Based on this, the present application also provides a possible implementation of the above distribution characterization function. Under this implementation, the distribution characterization function is specifically used to describe the distribution state of the grayscale values ​​in the to-be-processed area; and the determination process of the distribution characterization function may include: first, according to a preset conversion formula (for example, formula (39) below), convert each pixel value in the to-be-processed area in the kth image to be used above into a grayscale value to obtain the grayscale value at each pixel position in the to-be-processed area; and then use these grayscale values ​​to construct the distribution characterization function corresponding to the kth image to be used, and the distribution characterization function is used to describe the distribution state of the grayscale values ​​in the to-be-processed area.

[0290] Gray j =w R ×R j +w G ×G j +w B ×B j (39)

[0291] In the formula, (R j ,G j ,B j ) represents the pixel value of the jth pixel position in the to-be-processed area in the kth image to be used above, j is a positive integer, j≤J, J is a positive integer, and J represents the number of pixels in the to-be-processed area in the kth image to be used; R j G represents the red channel value in the pixel value of the j-th pixel position; j represents the green channel value of the pixel value at the j-th pixel position; B j represents the blue channel value in the pixel value of the j-th pixel position; w R represents the weight corresponding to the red channel, and w R =0.2989; w G represents the weight corresponding to the green channel, and w G =0.5870; w B represents the weight corresponding to the blue channel, and w B =0.1140; Grayj Represents the grayscale value of the j-th pixel position.

[0292] Based on the relevant content of step 43 above, it can be known that in some application scenarios, for the kth image to be used above, since the kth image to be used is collected according to the second wide angle, so that the pixel values ​​recorded in the kth image to be used are all the pixel values ​​required to be referenced when constructing the fisheye image, in order to overcome the above stitching problem as much as possible, the distribution characterization function corresponding to the kth image to be used can be constructed based on the pixel values ​​in the area to be processed in the kth image to be used, so that the distribution characterization function is used to describe the distribution state of the grayscale values ​​in the area to be processed, so that the grayscale value dilution processing can be performed on the pixel values ​​in the area to be processed in the kth image to be used based on the distribution characterization function.

[0293] Step 44: For any pixel position in the to-be-processed area in the kth image to be used above, the pixel value at the pixel position in the kth image to be used is updated using the weighted processing result between the pixel value at the pixel position in the kth image to be used and at least one reference pixel value corresponding to the pixel value; the weighted weight of the at least one reference pixel value is determined based on the function value corresponding to the distribution representation function corresponding to the pixel value in the kth image to be used, and the distance between the position where the reference pixel value is located in the kth image to be used and the pixel position satisfies the preset distance condition. Wherein, k is a positive integer, k≤K.

[0294] Wherein, for the j-th pixel position in the to-be-processed area in the k-th image to be used above, at least one reference pixel value corresponding to the pixel value at the j-th pixel position refers to the pixel values ​​at some pixel positions that exist in the k-th image to be used and are closer to the j-th pixel position, so that the at least one reference pixel value can represent the neighboring pixel values ​​of the pixel value at the j-th pixel position. Wherein, j is a positive integer, j≤J, J is a positive integer, and J represents the number of pixels in the to-be-processed area in the k-th image to be used.

[0295] In addition, the present application does not limit the number of reference pixel values ​​in the at least one reference pixel value corresponding to the pixel value at the j-th pixel position above, for example, it can be one or more.

[0296] In addition, the present application does not limit the determination process of at least one reference pixel value corresponding to the pixel value at the j-th pixel position above. For example, it can be specifically as follows: for the j-th pixel position in the to-be-processed area in the k-th image to be used above, a pixel position whose distance from the j-th pixel position satisfies a preset distance condition is searched from the k-th image to be used above as a reference pixel position, and the pixel value at the reference pixel position is determined as the reference pixel value corresponding to the pixel value at the j-th pixel position. The preset distance condition can be pre-set, for example, the preset distance condition can specifically be: the distance is less than a preset distance threshold. For another example, after the pixel positions other than the j-th pixel position in the k-th image to be used are sorted from near to far according to the distance between them and the j-th pixel position, the preset distance condition can be: the arrangement position number is less than or equal to the preset position number.

[0297] Based on the content of the previous paragraph, it can be known that for the j-th pixel position in the to-be-processed area in the k-th image to be used above, the distance between the position of each reference pixel value in at least one reference pixel value corresponding to the pixel value at the j-th pixel position in the k-th image to be used and the j-th pixel position satisfies the preset distance condition, so that these reference pixel values ​​can represent pixel values ​​that are adjacent to the pixel value at the j-th pixel position.

[0298] In addition, for the j-th pixel position in the area to be processed in the k-th image to be used above, the weighted weight of at least one reference pixel value corresponding to the pixel value at the j-th pixel position can be determined based on the function value corresponding to the pixel value at the j-th pixel position in the distribution representation function corresponding to the k-th image to be used; and the present application does not limit the determination method, for example, it can be specifically: first use the pixel value at the j-th pixel position as the input parameter of the distribution representation function corresponding to the k-th image to be used, and obtain the function value corresponding to the pixel value at the j-th pixel position, so that the function value can represent the probability of occurrence of the pixel value at the j-th pixel position in the area to be processed in the k-th image to be used; then determine the function value corresponding to the pixel value at the j-th pixel position as the weighted weight of the at least one reference pixel value, so that weighted processing can be performed on the at least one reference pixel value based on the weighted weight later.

[0299] Furthermore, the present application does not limit the pixel value update process of the j-th pixel position in the to-be-processed area in the k-th image to be used above. For example, in order to achieve fading processing while retaining the pixel color as much as possible, a grayscale value fading processing method (for example, the grayscale value fading processing method shown in the above formula (39) and the following formulas (40)-(44)) can be used to update the pixel value of the j-th pixel position.

[0300]

[0301]

[0302] weight j =F(Gray j ) (42)

[0303]

[0304]

[0305] In the formula, represents the b-th reference pixel value corresponding to the pixel value at the j-th pixel position above, where b is a positive integer, 1≤b≤B, and B represents the number of reference pixel values ​​in at least one reference pixel value corresponding to the pixel value at the j-th pixel position; represents the red channel value in the b-th reference pixel value corresponding to the pixel value at the j-th pixel position; Indicates the green channel value in the b-th reference pixel value corresponding to the pixel value at the j-th pixel position; represents the blue channel value in the bth reference pixel value corresponding to the pixel value at the jth pixel position; R represents the weight corresponding to the red channel, and w R =0.2989; w G represents the weight corresponding to the green channel, and w G =0.5870; w B represents the weight corresponding to the blue channel, and w B =0.1140; represents the gray value obtained by converting the bth reference pixel value corresponding to the pixel value at the jth pixel position; represents the average gray value; represents the updated gray value of the j-th pixel position; represents the updated pixel value of the j-th pixel position; G represents the red channel value in the updated pixel value of the j-th pixel position; j Indicates the green channel value of the jth pixel position above; B j represents the blue channel value of the j-th pixel position; F(·) refers to the distribution characterization function corresponding to the k-th image to be used, and F(·) is used to represent the distribution state of the grayscale value in the area to be processed.

[0306] It should be noted that, relative to the pixel value before update (R j ,G j ,B j), the updated pixel value at the j-th pixel position In the above example, only the red channel value is updated, that is, the value of R j Updated to However, keeping the values ​​of the other two color channels unchanged can effectively avoid color changes caused by adjusting the brightness and darkness.

[0307] Based on the grayscale value fading process shown in the above formulas (39)-(44), it can be known that in some application scenarios, for the pixel value of the jth pixel position in the to-be-processed area in the above kth image to be used, the pixel value of the jth pixel position can be first converted into a grayscale value to obtain the grayscale value of the jth pixel position, and each reference pixel value corresponding to the pixel value of the jth pixel position is converted into a grayscale value to obtain each reference grayscale value corresponding to the grayscale value of the jth pixel position; then, the average value between all reference grayscale values ​​corresponding to the grayscale value of the jth pixel position is calculated to obtain the grayscale reference average value corresponding to the grayscale value of the jth pixel position; then, the weighted sum value between the grayscale reference average value and the grayscale value of the jth pixel position is used to determine the updated grayscale value of the jth pixel position; finally, the updated grayscale value of the jth pixel position is converted into RGB to obtain the updated pixel value of the jth pixel position.

[0308] Based on the relevant content of step 44 above, it can be known that in some application scenarios, for the kth image to be used above, the low brightness defect in the area to be processed can be overcome by diluting the grayscale value of the pixel values ​​at each pixel position in the area to be processed in the kth image to be used.

[0309] Step 45: Generate a fisheye image based on the position mapping relationship and at least two images to be used, wherein the pixel values ​​in the fisheye image are determined based on the pixel values ​​in the at least two images to be used, and the fisheye image is used to describe the state of the target object under at least two viewing angles.

[0310] It should be noted that the implementation of step 45 is similar to the implementation of S3 above, and for the sake of brevity, it will not be repeated here.

[0311] Based on the relevant contents of steps 41 to 45 above, it can be known that in some application scenarios, in order to overcome the above-mentioned stitching problem, the pixel values ​​in the edge area of ​​the acquired image data can be firstly faded with grayscale values ​​to smooth the light and dark transition state in the image data, so that the light and dark transition of all areas of the final faded image is relatively smooth, and thus there will be no stitching problem in the fisheye image constructed based on the faded image.

[0312] In addition, the embodiments of the present application do not limit the execution subject of the image generation method provided in the embodiments of the present application. For example, the image generation method provided in the embodiments of the present application can be applied to a terminal device or a server. For another example, the image generation method provided in the embodiments of the present application can also be implemented with the help of the data interaction process between the terminal device and the server. Among them, the terminal device can be a smart phone, a computer, a personal digital assistant (PDA), a tablet computer, etc. The server can be a stand-alone server, a cluster server, or a cloud server.

[0313] In addition, the image generation method provided in the embodiment of the present application may have the advantages shown in ①-⑤ below:

[0314] ① This application uses a two-step mapping method to realize the acquisition of image data (for example, Figure 2 The six square images shown in the figure are spliced ​​into a fisheye image (for example, a 2:1 fisheye image), so that the image generation method provided in the present application is suitable for constructing any fisheye image.

[0315] ② This application provides two ways to fade the seams (for example, one is to set the image acquisition wide angle to 120 degrees, but only map the pixel values ​​within the wide angle range of 90 degrees in the acquired image data to the fisheye image; the other is to perform natural light and dark filtering on the stitching edges by fading the grayscale value), which can effectively avoid defects caused by the seam problem.

[0316] ③The image generation method provided in this application can achieve a relatively high fisheye image construction speed (for example, the construction time of an 8K fisheye image is approximately 2 seconds).

[0317] ④ The image generation method provided in this application has strong compatibility and is suitable for various rendering scenarios (for example, rasterization rendering scenes, ray tracing rendering scenes, path rendering scenes, etc.).

[0318] ⑤ The image generation method provided in this application shows strong compatibility in terms of computer models, so that the image generation method provided in this application can be run on any computer configuration.

[0319] Based on the image generation method provided in the embodiment of the present application, the embodiment of the present application also provides an image generation device. Figure 5 Explain and illustrate. Figure 5 This is a schematic diagram of the structure of an image generating device provided in an embodiment of the present application. It should be noted that for the technical details of the image generating device provided in an embodiment of the present application, please refer to the relevant contents of the image generating method above.

[0320] like Figure 5As shown, the image generating device 500 provided in the embodiment of the present application includes:

[0321] An acquisition unit 501 is used to acquire at least two images to be used, where different images to be used are used to describe the states of the target object at different viewing angles;

[0322] A construction unit 502 is used to construct a position mapping relationship, where the position mapping relationship is used to describe the correspondence between the pixel positions in the at least two images to be used and the pixel positions in the fisheye image;

[0323] The generation unit 503 is used to generate the fisheye image based on the position mapping relationship and the at least two images to be used, wherein the pixel values ​​in the fisheye image are determined based on the pixel values ​​in the at least two images to be used, and the fisheye image is used to describe the state of the target object under at least two viewing angles.

[0324] In one possible implementation, the position mapping relationship includes a first sub-mapping relationship and a second sub-mapping relationship; the first sub-mapping relationship is used to describe the correspondence between the pixel position in the fisheye image and the three-dimensional coordinate position in the three-dimensional sphere; the second sub-mapping relationship is used to describe the correspondence between the three-dimensional coordinate position in the three-dimensional sphere and the pixel position in the at least two images to be used.

[0325] In a possible implementation manner, the construction unit 502 includes:

[0326] A first determining subunit is configured to determine, for any three-dimensional coordinate position in the three-dimensional spherical surface, a target image corresponding to the three-dimensional coordinate position from the at least two images to be used according to the preset mapping conditions of the images to be used; the three-dimensional coordinate position satisfies the preset mapping conditions of the target image;

[0327] A second determining subunit is used to determine the mapping pixel position corresponding to the three-dimensional coordinate position from the target image;

[0328] The third determining subunit is used to determine the second sub-mapping relationship according to the corresponding relationship between the three-dimensional coordinate position and the mapping pixel position, where the second sub-mapping relationship includes the corresponding relationship between the three-dimensional coordinate position and the mapping pixel position.

[0329] In a possible implementation manner, for any image to be used among the at least two images to be used, the preset mapping condition of the image to be used is determined according to a capture viewing angle of the image to be used.

[0330] In a possible implementation manner, the target image is used to describe the state of the target object from a top surface perspective, and the preset mapping conditions of the target image are: the data value in the third dimension of the three-dimensional coordinate position is greater than 0, the absolute value of the data value in the third dimension of the three-dimensional coordinate position is not less than the absolute value of the data value in the first dimension of the three-dimensional coordinate position, and the absolute value of the data value in the third dimension of the three-dimensional coordinate position is not less than the absolute value of the data value in the second dimension of the three-dimensional coordinate position;

[0331] or,

[0332] The target image is used to describe the state of the target object from the bottom surface perspective, and the preset mapping conditions of the target image are: the data value in the third dimension of the three-dimensional coordinate position is less than 0, the absolute value of the data value in the third dimension of the three-dimensional coordinate position is not less than the absolute value of the data value in the first dimension of the three-dimensional coordinate position, and the absolute value of the data value in the third dimension of the three-dimensional coordinate position is not less than the absolute value of the data value in the second dimension of the three-dimensional coordinate position;

[0333] or,

[0334] The target image is used to describe the state of the target object in the frontal perspective, and the preset mapping conditions of the target image are: the data value in the first dimension of the three-dimensional coordinate position is greater than 0, the absolute value of the data value in the first dimension of the three-dimensional coordinate position is not less than the absolute value of the data value in the second dimension of the three-dimensional coordinate position, and the absolute value of the data value in the first dimension of the three-dimensional coordinate position is not less than the absolute value of the data value in the third dimension of the three-dimensional coordinate position;

[0335] or,

[0336] The target image is used to describe the state of the target object from a rear perspective, and the preset mapping conditions of the target image are: the data value in the first dimension of the three-dimensional coordinate position is less than 0, the absolute value of the data value in the first dimension of the three-dimensional coordinate position is not less than the absolute value of the data value in the second dimension of the three-dimensional coordinate position, and the absolute value of the data value in the first dimension of the three-dimensional coordinate position is not less than the absolute value of the data value in the third dimension of the three-dimensional coordinate position;

[0337] or,

[0338] The target image is used to describe the state of the target object in the right viewing angle, and the preset mapping conditions of the target image are: the data value in the second dimension of the three-dimensional coordinate position is greater than 0, the absolute value of the data value in the second dimension of the three-dimensional coordinate position is not less than the absolute value of the data value in the first dimension of the three-dimensional coordinate position, and the absolute value of the data value in the second dimension of the three-dimensional coordinate position is not less than the absolute value of the data value in the third dimension of the three-dimensional coordinate position;

[0339] or,

[0340] The target image is used to describe the state of the target object in the left viewing angle, and the preset mapping conditions of the target image are: the data value in the second dimension of the three-dimensional coordinate position is less than 0, the absolute value of the data value in the second dimension of the three-dimensional coordinate position is not less than the absolute value of the data value in the first dimension of the three-dimensional coordinate position, and the absolute value of the data value in the second dimension of the three-dimensional coordinate position is not less than the absolute value of the data value in the third dimension of the three-dimensional coordinate position.

[0341] In one possible implementation, the second determination subunit is specifically used to determine a mapping pixel position corresponding to the three-dimensional coordinate position based on a preset mapping algorithm corresponding to the target image, wherein the preset mapping algorithm is determined based on a capture viewing angle of the target image.

[0342] In a possible implementation manner, the second determining subunit includes:

[0343] A fourth determining subunit is used to determine a data compression rate according to a data value in a target dimension in the three-dimensional coordinate position; the target dimension is determined according to a capture viewing angle of the target image;

[0344] The fifth determination subunit is used to determine the mapping pixel position corresponding to the three-dimensional coordinate position according to the data compression rate and other two data values ​​in the three-dimensional coordinate position except the data value in the target dimension.

[0345] In one possible implementation, the fifth determination subunit is specifically used to: determine a theoretical mapping position corresponding to the three-dimensional coordinate position based on the data compression rate and the other two data values ​​in the three-dimensional coordinate position except the data value in the target dimension; determine at least one adjacent pixel position corresponding to the theoretical mapping position from the target image; and determine a mapping pixel position corresponding to the three-dimensional coordinate position based on the at least one adjacent pixel position.

[0346] In one possible implementation, the mapping pixel position corresponding to the three-dimensional coordinate position includes the at least one adjacent pixel position; the pixel value at the pixel position corresponding to the three-dimensional coordinate position in the fisheye image is obtained by weighted summing the pixel values ​​at the at least one adjacent pixel position in the target image; for any adjacent pixel position, the weighted weight corresponding to the pixel value at the adjacent pixel position in the target image is determined based on the distance between the adjacent pixel position and the theoretical mapping position.

[0347] In a possible implementation manner, the at least one adjacent pixel position includes four adjacent pixel positions.

[0348] In one possible implementation, the at least two perspectives include K perspectives, where K is a positive integer; the at least two images to be used include K images to be used, and the kth image to be used is obtained by capturing the target object according to the kth perspective, so that the kth image to be used is used to describe the state of the target object at the kth perspective, and k is a positive integer, k≤K.

[0349] In a possible implementation manner, the kth image to be used is obtained by capturing the target object according to the kth viewing angle and the first wide angle;

[0350] The generating unit 503 includes:

[0351] a sixth determining subunit, configured to determine, according to an image size of the kth image to be used, image region description information corresponding to a second wide angle in the kth image to be used; the second wide angle is smaller than the first wide angle, k is a positive integer, k≤K;

[0352] a seventh determining subunit, configured to extract pixel values ​​from the kth image to be used according to the image region description information corresponding to the second wide angle in the kth image to be used, so as to obtain the pixel values ​​corresponding to the second wide angle in the kth image to be used, where k is a positive integer, k≤K;

[0353] The eighth determination subunit is used to generate the fisheye image according to the position mapping relationship and the pixel values ​​corresponding to the second wide angle in the K images to be used.

[0354] In a possible implementation manner, the kth image to be used is obtained by capturing the target object according to the kth viewing angle and the second wide angle;

[0355] The image generating device 500 further includes:

[0356] An analysis unit, configured to construct a distribution characterization function according to pixel values ​​in the to-be-processed area in the k-th image to be used, wherein the distribution characterization function is used to describe a brightness distribution state in the to-be-processed area;

[0357] An updating unit is used to update the pixel value at any pixel position in the to-be-processed area in the kth image to be used by using a weighted processing result between the pixel value at the pixel position in the kth image to be used and at least one reference pixel value corresponding to the pixel value; the weighted weight of the at least one reference pixel value is determined based on the function value corresponding to the pixel value in the distribution characterization function, and the distance between the position of the reference pixel value in the kth image to be used and the pixel position satisfies a preset distance condition.

[0358] In a possible implementation manner, for any pixel position in the to-be-processed area in the k-th image to be used, the grayscale value of the pixel position is determined according to the pixel value at the pixel position in the k-th image to be used;

[0359] The distribution characterization function is used to describe the gray value distribution state presented in the area to be processed;

[0360] The updating unit is specifically used to update the pixel value at the pixel position in the kth image to be used by utilizing the weighted processing result between the grayscale value of the pixel position in the kth image to be used and at least one reference grayscale value corresponding to the grayscale value; the at least one reference grayscale value is determined based on the at least one reference pixel value.

[0361] In a possible implementation manner, the target object is a three-dimensional house description object.

[0362] In a possible implementation manner, the at least two viewing angles include a top viewing angle, a bottom viewing angle, a front viewing angle, a back viewing angle, a left viewing angle, and a right viewing angle.

[0363] Based on the relevant contents of the above-mentioned image generating device 500, it can be known that for the image generating device 500 provided in the embodiment of the present application, at least two images to be used are first obtained (for example, square images collected from 6 orthogonal viewing angles for the target object, etc.), so that different images to be used are used to describe the state of the target object at different viewing angles, so that the at least two images to be used can represent the state of the target object at at least two viewing angles; then a position mapping relationship is constructed, so that the position mapping relationship is used to describe the correspondence between the pixel positions in the at least two images to be used and the pixel positions in the fisheye image; finally, the fisheye image is generated based on the position mapping relationship and the at least two images to be used, so that the purpose of obtaining the fisheye image can be achieved. Among them, because the pixel values ​​in the fisheye image are determined based on the pixel values ​​in the at least two images to be used, the fisheye image can describe the state of the target object at at least two viewing angles, so that the fisheye image can better describe the target object, which is conducive to improving the effect of obtaining the fisheye image.

[0364] In addition, an embodiment of the present application also provides an electronic device, which includes a processor and a memory: the memory is used to store instructions or computer programs; the processor is used to execute the instructions or computer programs in the memory, so that the electronic device executes any implementation of the image generation method provided in the embodiment of the present application.

[0365] See also Figure 6 , which shows a schematic diagram of the structure of an electronic device 600 suitable for implementing the embodiment of the present disclosure. The terminal device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0366] like Figure 6 As shown, the electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0367] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6 The electronic device 600 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0368] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 609, or installed from the storage device 608, or installed from the ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0369] The electronic device provided by the embodiment of the present disclosure and the method provided by the above embodiment belong to the same inventive concept, and the technical details not fully described in this embodiment can be referred to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0370] The embodiment of the present application also provides a computer-readable medium, in which instructions or computer programs are stored. When the instructions or computer programs are executed on a device, the device executes any implementation of the image generation method provided in the embodiment of the present application.

[0371] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0372] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (Hyper Text Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0373] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0374] The computer-readable medium carries one or more programs, and when the one or more programs are executed by the electronic device, the electronic device can execute the method.

[0375] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0376] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0377] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit / module does not, in some cases, constitute a limitation on the unit itself.

[0378] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0379] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0380] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the system or device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0381] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0382] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0383] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0384] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image generation method, characterized in that: The method comprises: Acquire at least two images to be used, where different images to be used are used to describe states of the target object at different viewing angles; Constructing a position mapping relationship, where the position mapping relationship is used to describe the correspondence between the pixel positions in the at least two images to be used and the pixel positions in the fisheye image; The fisheye image is generated based on the position mapping relationship and the at least two images to be used. The pixel values ​​in the fisheye image are determined based on the pixel values ​​in the at least two images to be used. The fisheye image is used to describe the state of the target object under at least two viewing angles.

2. The method according to claim 1, characterized in that The position mapping relationship includes a first sub-mapping relationship and a second sub-mapping relationship; The first sub-mapping relationship is used to describe the correspondence between the pixel position in the fisheye image and the three-dimensional coordinate position in the three-dimensional spherical surface; The second sub-mapping relationship is used to describe the correspondence between the three-dimensional coordinate position in the three-dimensional spherical surface and the pixel position in the at least two images to be used.

3. The method according to claim 2, characterized in that The process of determining the second sub-mapping relationship includes: For any three-dimensional coordinate position in the three-dimensional spherical surface, according to the preset mapping conditions of each of the images to be used, a target image corresponding to the three-dimensional coordinate position is determined from the at least two images to be used; the three-dimensional coordinate position satisfies the preset mapping conditions of the target image; Determine a mapping pixel position corresponding to the three-dimensional coordinate position from the target image; The second sub-mapping relationship is determined according to the correspondence relationship between the three-dimensional coordinate position and the mapping pixel position, and the second sub-mapping relationship includes the correspondence relationship between the three-dimensional coordinate position and the mapping pixel position.

4. The method according to claim 3, characterized in that For any image to be used among the at least two images to be used, the preset mapping condition of the image to be used is determined according to the acquisition viewing angle of the image to be used.

5. The method according to claim 3, characterized in that: The target image is used to describe the state of the target object from the top view, and the preset mapping conditions of the target image are: the data value in the third dimension of the three-dimensional coordinate position is greater than 0, the absolute value of the data value in the third dimension of the three-dimensional coordinate position is not less than the absolute value of the data value in the first dimension of the three-dimensional coordinate position, and the absolute value of the data value in the third dimension of the three-dimensional coordinate position is not less than the absolute value of the data value in the second dimension of the three-dimensional coordinate position; or, The target image is used to describe the state of the target object from the bottom surface perspective, and the preset mapping conditions of the target image are: the data value in the third dimension of the three-dimensional coordinate position is less than 0, the absolute value of the data value in the third dimension of the three-dimensional coordinate position is not less than the absolute value of the data value in the first dimension of the three-dimensional coordinate position, and the absolute value of the data value in the third dimension of the three-dimensional coordinate position is not less than the absolute value of the data value in the second dimension of the three-dimensional coordinate position; or, The target image is used to describe the state of the target object in the frontal perspective, and the preset mapping conditions of the target image are: the data value in the first dimension of the three-dimensional coordinate position is greater than 0, the absolute value of the data value in the first dimension of the three-dimensional coordinate position is not less than the absolute value of the data value in the second dimension of the three-dimensional coordinate position, and the absolute value of the data value in the first dimension of the three-dimensional coordinate position is not less than the absolute value of the data value in the third dimension of the three-dimensional coordinate position; or, The target image is used to describe the state of the target object from a rear perspective, and the preset mapping conditions of the target image are: the data value in the first dimension of the three-dimensional coordinate position is less than 0, the absolute value of the data value in the first dimension of the three-dimensional coordinate position is not less than the absolute value of the data value in the second dimension of the three-dimensional coordinate position, and the absolute value of the data value in the first dimension of the three-dimensional coordinate position is not less than the absolute value of the data value in the third dimension of the three-dimensional coordinate position; or, The target image is used to describe the state of the target object in the right viewing angle, and the preset mapping conditions of the target image are: the data value in the second dimension of the three-dimensional coordinate position is greater than 0, the absolute value of the data value in the second dimension of the three-dimensional coordinate position is not less than the absolute value of the data value in the first dimension of the three-dimensional coordinate position, and the absolute value of the data value in the second dimension of the three-dimensional coordinate position is not less than the absolute value of the data value in the third dimension of the three-dimensional coordinate position; or, The target image is used to describe the state of the target object in the left viewing angle, and the preset mapping conditions of the target image are: the data value in the second dimension of the three-dimensional coordinate position is less than 0, the absolute value of the data value in the second dimension of the three-dimensional coordinate position is not less than the absolute value of the data value in the first dimension of the three-dimensional coordinate position, and the absolute value of the data value in the second dimension of the three-dimensional coordinate position is not less than the absolute value of the data value in the third dimension of the three-dimensional coordinate position.

6. The method according to claim 3, characterized in that Determining the mapping pixel position corresponding to the three-dimensional coordinate position from the target image includes: The mapping pixel position corresponding to the three-dimensional coordinate position is determined according to a preset mapping algorithm corresponding to the target image, wherein the preset mapping algorithm is determined according to the acquisition viewing angle of the target image.

7. The method according to claim 6, characterized in that The preset mapping algorithm corresponding to the target image includes: Determining a data compression rate according to a data value in a target dimension in the three-dimensional coordinate position; the target dimension is determined according to a capture viewing angle of the target image; The mapping pixel position corresponding to the three-dimensional coordinate position is determined according to the data compression rate and other two data values ​​in the three-dimensional coordinate position except the data value in the target dimension.

8. The method according to claim 7, characterized in that The determining, based on the data compression rate and two other data values ​​in the three-dimensional coordinate position except the data value in the target dimension, a mapping pixel position corresponding to the three-dimensional coordinate position includes: Determine a theoretical mapping position corresponding to the three-dimensional coordinate position according to the data compression rate and two other data values ​​in the three-dimensional coordinate position except the data value in the target dimension; Determine at least one adjacent pixel position corresponding to the theoretical mapping position from the target image; A mapping pixel position corresponding to the three-dimensional coordinate position is determined according to the at least one adjacent pixel position.

9. The method according to claim 8, characterized in that The mapped pixel position corresponding to the three-dimensional coordinate position includes the at least one adjacent pixel position; The pixel value at the pixel position corresponding to the three-dimensional coordinate position in the fisheye image is obtained by weighted summing the pixel values ​​at the at least one adjacent pixel position in the target image; For any adjacent pixel position, the weighted weight corresponding to the pixel value at the adjacent pixel position in the target image is determined according to the distance between the adjacent pixel position and the theoretical mapping position.

10. The method according to claim 8, characterized in that The at least one adjacent pixel position includes 4 adjacent pixel positions.

11. The method according to claim 1, characterized in that: The at least two viewing angles include K viewing angles, where K is a positive integer; The at least two images to be used include K images to be used, and the kth image to be used is obtained by capturing the target object according to the kth viewing angle, so that the kth image to be used is used to describe the state of the target object at the kth viewing angle, k is a positive integer, k≤K.

12. The method according to claim 11, characterized in that The kth image to be used is obtained by capturing the target object according to the kth viewing angle and the first wide angle; The step of generating the fisheye image according to the position mapping relationship and the at least two images to be used includes: Determining image region description information corresponding to a second wide angle in the kth image to be used according to an image size of the kth image to be used; The second wide angle is smaller than the first wide angle, k is a positive integer, k≤K; Extracting pixel values ​​from the kth image to be used according to the image region description information corresponding to the second wide angle in the kth image to be used, to obtain the pixel value corresponding to the second wide angle in the kth image to be used, where k is a positive integer, k≤K; The fisheye image is generated according to the position mapping relationship and the pixel values ​​corresponding to the second wide angle in the K images to be used.

13. The method according to claim 11, characterized in that The kth image to be used is obtained by capturing the target object according to the kth viewing angle and the second wide angle; Before generating the fisheye image according to the position mapping relationship and the at least two images to be used, the method further includes: Constructing a distribution characterization function according to pixel values ​​in the to-be-processed area in the kth to-be-used image, wherein the distribution characterization function is used to describe a brightness distribution state in the to-be-processed area; For any pixel position within the to-be-processed area in the kth image to be used, the pixel value at the pixel position in the kth image to be used is updated using a weighted processing result between the pixel value at the pixel position in the kth image to be used and at least one reference pixel value corresponding to the pixel value; the weighted weight of the at least one reference pixel value is determined based on a function value corresponding to the pixel value in the distribution characterization function, and the distance between the position of the reference pixel value in the kth image to be used and the pixel position satisfies a preset distance condition.

14. The method according to claim 13, characterized in that For any pixel position in the to-be-processed area in the k-th image to be used, the grayscale value of the pixel position is determined according to the pixel value at the pixel position in the k-th image to be used; The distribution characterization function is used to describe the gray value distribution state presented in the area to be processed; The updating of the pixel value at the pixel position in the kth image to be used by using a weighted processing result between the pixel value at the pixel position in the kth image to be used and at least one reference pixel value corresponding to the pixel value comprises: The pixel value at the pixel position in the kth image to be used is updated by utilizing the weighted processing result between the grayscale value of the pixel position in the kth image to be used and at least one reference grayscale value corresponding to the grayscale value; the at least one reference grayscale value is determined based on the at least one reference pixel value.

15. The method according to any one of claims 1 to 14, characterized in that: The target object is a three-dimensional house description object; and / or, The at least two viewing angles include a top viewing angle, a bottom viewing angle, a front viewing angle, a back viewing angle, a left viewing angle and a right viewing angle.

16. An image generating device, characterized in that: include: An acquisition unit, used to acquire at least two images to be used, where different images to be used are used to describe the states of the target object at different viewing angles; A construction unit, used to construct a position mapping relationship, wherein the position mapping relationship is used to describe the correspondence between the pixel positions in the at least two images to be used and the pixel positions in the fisheye image; A generating unit is used to generate the fisheye image according to the position mapping relationship and the at least two images to be used, wherein the pixel values ​​in the fisheye image are determined according to the pixel values ​​in the at least two images to be used, and the fisheye image is used to describe the state of the target object under at least two viewing angles.

17. An electronic device, characterized in that: The device comprises: a processor and a memory; The memory is used to store instructions or computer programs; The processor is used to execute the instructions or computer programs in the memory so that the electronic device executes the method according to any one of claims 1 to 15.

18. A computer readable medium, characterized in that The computer-readable medium stores instructions or computer programs, and when the instructions or computer programs are executed on a device, the device executes the method according to any one of claims 1 to 15.

Images