Image processing method and device

Abstract

The embodiment of the invention provides a picture processing method and device. The picture processing method comprises the steps that a target picture is obtained; obtaining a reference picture, the shooting position of the target picture is the same as the shooting position of the reference picture, the shooting orientation of the target picture is the same as the shooting orientation of the reference picture, and the field angle of view of the target picture is smaller than the field angle of view of the reference picture; generating a display picture based on a target model, the reference picture and the target picture; wherein the size of the displayed picture is larger than that of the target picture, the displayed picture comprises picture content of the target picture and expanded content, and the expanded content is display content generated based on a target model, the reference picture and the target picture and used for expanding the target picture in at least one direction. According to the technical scheme, the image expansion accuracy can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method and device. Background Art

[0002] The current technical solution for image expansion usually directly uses generative artificial intelligence to directly process the images that need to be expanded, but the extended content of the generated extended images does not conform to common sense in reality, and thus there is still a problem of low accuracy in image expansion. Summary of the invention

[0003] In view of this, the embodiments of the present application at least provide an image processing method and device.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] In a first aspect, an embodiment of the present application provides an image processing method, comprising: obtaining a target image; obtaining a reference image, wherein a shooting position of the target image is the same as a shooting position of the reference image, a shooting orientation of the target image is the same as a shooting orientation of the reference image, and a field of view angle of the target image is smaller than a field of view angle of the reference image; generating a display image based on a target model, a reference image, and a target image; wherein a size of the display image is larger than a size of the target image, and the display image includes image content of the target image and extended content, and the extended content is display content generated based on the target model, the reference image, and the target image for expanding at least one direction of the target image.

[0006] In a second aspect, an embodiment of the present application provides an image processing method, comprising: in response to a photo taking instruction, obtaining a target image and a corresponding reference image including the same object; the shooting position of the target image is the same as the shooting position of the reference image, the shooting orientation of the target image is the same as the shooting orientation of the reference image, and the field of view angle of the target image is smaller than the field of view angle of the reference image; saving the reference image and the target image; displaying the target image; the reference image is used to respond to an extension instruction for the target image, and together with the target image, a display image is generated based on a target model, the size of the display image is larger than the size of the target image, and the display image includes the image content of the target image and the extended content.

[0007] In a third aspect, an embodiment of the present application provides an image processing device, comprising: a first acquisition module, used to obtain a target picture; a second acquisition module, used to obtain a reference picture; the shooting position of the target picture is the same as the shooting position of the reference picture, the shooting direction of the target picture is the same as the shooting direction of the reference picture, and the field of view angle of the target picture is smaller than the field of view angle of the reference picture; a generation module, used to generate a display picture based on a target model, a reference picture and a target picture; wherein the size of the display picture is larger than the size of the target picture, the display picture includes the picture content of the target picture and extended content, and the extended content is generated based on the target model, the reference picture and the target picture to expand the display content in at least one direction of the target picture.

[0008] In a fourth aspect, an embodiment of the present application provides an image processing device, comprising: a third acquisition module, for responding to a photo taking instruction, to obtain a target image and a corresponding reference image including the same object; the shooting position of the target image is the same as the shooting position of the reference image, the shooting orientation of the target image is the same as the shooting orientation of the reference image, and the field of view angle of the target image is smaller than the field of view angle of the reference image; a saving module, for saving the reference image and the target image; a display module, for displaying the target image; the reference image is used to respond to an extension instruction for the target image, and generate a display image together with the target image based on a target model, the size of the display image is larger than the size of the target image, and the display image includes the image content of the target image and the extended content.

[0009] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising a processing device and a display screen, wherein the image processing device comprises: a first acquisition component, used to obtain a target image; a second acquisition component, used to obtain a reference image, the shooting position of the target image is the same as the shooting position of the reference image, the shooting direction of the target image is the same as the shooting direction of the reference image, and the field of view angle of the target image is smaller than the field of view angle of the reference image; a generation component, used to generate a display image based on a target model, a reference image and a target image; wherein the size of the display image is larger than the size of the target image, the display image includes the image content of the target image and extended content, and the extended content is generated based on the target model, the reference image and the target image to expand the display content in at least one direction of the target image; the display screen is used to display the target image.

[0010] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which implements some or all of the steps in the above method when executed by a processor.

[0011] In a seventh aspect, an embodiment of the present application provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement some or all of the steps in the above method.

[0012] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0014] Figure 1 A schematic diagram of an implementation flow of an image processing method provided in an embodiment of the present application;

[0015] Figure 2 A schematic diagram of an implementation flow of an image processing method provided in an embodiment of the present application;

[0016] Figure 3 A schematic diagram of an implementation flow of an image processing method provided in an embodiment of the present application;

[0017] Figure 4A A schematic diagram of an implementation flow of an image processing method provided in an embodiment of the present application;

[0018] Figure 4B A schematic diagram of an implementation flow of an image processing method provided in an embodiment of the present application;

[0019] Figure 5 A schematic diagram of an implementation flow of an image processing method provided in an embodiment of the present application;

[0020] Figure 6 A schematic diagram of an implementation flow of an image processing method provided in an embodiment of the present application;

[0021] Figure 7 A schematic diagram of an implementation flow of an image processing method provided in an embodiment of the present application;

[0022] Figure 8 A schematic diagram of an implementation flow of an image processing method provided in an embodiment of the present application;

[0023] Fig. 9 A schematic diagram of an implementation flow of an image processing method provided in an embodiment of the present application;

[0024] Fig.10 A schematic diagram of an implementation flow of an image processing method provided in an embodiment of the present application;

[0025] Fig.11 A schematic diagram of an implementation flow of an image processing method provided in an embodiment of the present application;

[0026] Fig.12A schematic diagram of an implementation method of a map expansion method provided in an embodiment of the present application;

[0027] Fig.13 A schematic diagram of an implementation flow of a method for expanding a map provided in an embodiment of the present application;

[0028] Fig.14 A schematic diagram of implementing wide-angle image storage provided in an embodiment of the present application;

[0029] Fig.15 A schematic diagram of an implementation method of a map expansion method provided in an embodiment of the present application;

[0030] Fig.16A A schematic diagram of an implementation of a shooting principle provided in an embodiment of the present application;

[0031] Fig. 16B A schematic diagram of an implementation of a shooting principle provided in an embodiment of the present application;

[0032] Fig.17 A schematic diagram of an implementation flow of a method for expanding a map provided in an embodiment of the present application;

[0033] Fig.18A A schematic diagram of an implementation of a shooting end architecture provided in an embodiment of the present application;

[0034] Fig.18B A schematic diagram of implementing an extended image provided in an embodiment of the present application

[0035] Fig.19 A schematic diagram of an implementation method of a map expansion method provided in an embodiment of the present application;

[0036] Fig. 20 A schematic diagram of an implementation method of a map expansion method provided in an embodiment of the present application;

[0037] Fig.21 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application;

[0038] Fig. 22 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application;

[0039] Fig.23 A hardware entity diagram of an electronic device provided in an embodiment of the present application

[0040] Fig.24 A hardware entity schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions of the present application are further elaborated in detail below in conjunction with the drawings and embodiments. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0042] In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments, but it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict. The terms "first / second / third" involved are merely used to distinguish similar objects and do not represent a specific order for the objects. It is understood that "first / second / third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0043] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing this application and are not intended to limit this application.

[0044] The current technical solution for image expansion usually directly uses generative artificial intelligence to directly process the images that need to be expanded, but the extended content of the generated extended images does not conform to common sense in reality, and thus there is still a problem of low accuracy in image expansion.

[0045] In response to the above technical problems, an embodiment of the present application provides an image processing method. Since the shooting direction and shooting position of the reference picture are the same as those of the target picture, it can be determined that the picture content in the reference picture has the same perspective and scene as the picture content in the target picture; based on the field of view angle of the reference picture being larger than the field of view angle of the target picture, it can be determined that the field of view range of the picture content in the reference picture is larger than the field of view range of the picture content in the target picture. Therefore, by expanding the picture content in the target picture based on the picture content in the reference picture that conforms to common sense in reality, a display picture that conforms to common sense in reality can be obtained, thereby improving the accuracy of picture expansion.

[0046] Figure 1 The following is a schematic diagram of an implementation flow of an image processing method provided in an embodiment of the present application. Figure 1 As shown, the method includes the following steps S101 to S103, combining Figure 1 The steps shown are explained.

[0047] Step S101, obtaining a target image.

[0048] In some embodiments, the target image represents an image that needs to be expanded, and the target image is obtained in response to the expansion operation on the target image.

[0049] In some embodiments, the target image may be obtained by the following implementations: obtained by photographing with any photographing device, obtained from a photographed and saved image, and the like.

[0050] Exemplarily, the target image may be an image captured by a mobile phone or a camera; or may be obtained from a photo album of a mobile phone or a storage card of a camera.

[0051] Step S102, obtaining a reference picture, wherein the shooting position of the target picture is the same as that of the reference picture, the shooting direction of the target picture is the same as that of the reference picture, and the field of view angle of the target picture is smaller than that of the reference picture.

[0052] In some embodiments, the reference picture is used to expand the target picture based on the content of the reference picture that is not in the target picture when performing an expansion operation on the content in the target picture.

[0053] Exemplarily, the target picture includes 60% of object A and the reference picture includes 100% of object A. The target picture is expanded based on the remaining part of object A in the reference picture except the 60% of object A in the target picture to obtain an extended picture; the viewing angle of the extended picture is the same as that of the target picture, and the completeness of object A in the extended picture is greater than that in the target picture.

[0054] In some embodiments, obtaining the reference image can be achieved through the following implementations: photographing it with any photographing device, obtaining it from a saved image, etc.

[0055] In some embodiments, the reference image may be stored in a header file of the target image; it is understandable that, in response to an instruction to expand the target image, the reference image may also be obtained by parsing the header file of the target image.

[0056] In some embodiments, there is an association relationship between the reference picture and the target picture; in response to an instruction to expand the target picture, the reference picture is obtained from the saved pictures based on the association relationship between the reference picture and the target picture.

[0057] In some embodiments, the target image shooting end position is the same as the reference image shooting end position; it can be understood that the position of the shooting device shooting the target image is the same as the position of the shooting device shooting the reference image. Exemplarily, the target image is obtained by shooting the target object at point A by the shooting device; the reference image is obtained by shooting the target object at point A by the shooting device.

[0058] In some embodiments, the orientation of the target image is the same as the orientation of the reference image; it can be understood that in the process of photographing the target object by a photographing device to obtain the target image and the reference image, the shooting angle of the lens of the photographing device relative to the target object is the same; illustratively, the photographing device photographs the target object at an upward angle of 30 degrees in the east direction of the target object to obtain the target image; similarly, the photographing device photographs the target object at an upward angle of 30 degrees in the east direction of the target object to obtain the reference image.

[0059] In some embodiments, the field of view angle of the target image is smaller than the field of view angle of the reference image; it can be understood that the focal length of the lens of the shooting device for taking the target image is smaller than the focal length of the lens of the shooting device for taking the reference image, that is, the field of view range of the object in the target image is smaller than the field of view range of the object in the reference image.

[0060] In some embodiments, the target image and the reference image may be obtained by the same photographing device or by different photographing devices; wherein, if the target image and the reference image are obtained by the same photographing device, the target image may be obtained based on the main camera lens or telephoto lens of the photographing device, and the reference image may be obtained based on the wide-angle lens of the photographing device, and the field of view of the reference image obtained by the wide-angle lens is greater than that of the target image obtained by the main camera lens or telephoto lens.

[0061] Step S103, generating a display picture based on the target model, the reference picture and the target picture;

[0062] Among them, the size of the display picture is larger than the size of the target picture, and the display picture includes the picture content of the target picture and extended content, and the extended content is generated based on the target model, the reference picture and the target picture to expand the display content of at least one direction of the target picture.

[0063] In some embodiments, the target model is a trained picture extension model, and the trained picture extension model is trained in the following manner: obtaining multiple training samples; each training sample includes a first sample picture and a second sample picture, and a label picture corresponding to the training sample; inputting the first sample picture and the second sample picture into the picture extension model to be trained to obtain a predicted picture; adjusting the model parameters of the picture extension model to be trained based on the loss value between the predicted picture and the label picture until the convergence condition is met, and outputting the trained picture extension model.

[0064] In some embodiments, the target model may include a generative adversarial network model (GAN), a pixel recurrent neural network model (PRNN), etc.

[0065] In some embodiments, a display picture is generated based on the target model, the reference picture and the target picture; it can be understood that the target picture and the reference picture are input into a trained picture expansion model to obtain the display picture.

[0066] In some embodiments, the display picture includes the picture content of the target picture and the extended content. It can be understood that the completeness of the target content contained in the display picture is greater than the completeness of the target content in the target picture; exemplarily, the target picture contains 60% of the target object; the display picture contains 80% of the target object, wherein 20% of the target objects contained in the display picture are extended content; in other embodiments, the number or types of target content contained in the display picture is greater than the number or types of target content contained in the target picture; exemplarily, the display picture contains 2 trees and a person; the target picture contains a tree, wherein, except for the one tree in the target picture, the remaining one tree and one person in the display picture are extended content.

[0067] In some embodiments, extending at least one direction of the target picture may include: extending the target picture to the left, extending the target picture to the right, extending the target picture upward, extending the target picture downward, and the like.

[0068] In some embodiments, expanding the display content of at least one direction of the target image is determined based on the expansion requirement of the target image; exemplarily, the expansion requirement of the target image is to expand the target image by 30% to the left and 40% to the right, then the target image, the reference image and the expansion requirements of expanding the target image by 30% to the left and 40% to the right are input into the above-mentioned trained image expansion model to obtain a display image expanded by 30% to the left and 40% to the right, wherein, compared with the target image, the display image has the content expanded by 30% to the left and 40% to the right, which is the expanded content of the embodiment of the present application.

[0069] In some embodiments, the resolution of the display image is the same as the resolution of the target image. Therefore, after expanding the display image in at least one direction, the size of the display image obtained after expansion will increase while the resolution remains unchanged; it can be understood that the size corresponding to the expanded content is increased.

[0070] In an embodiment of the present application, based on the fact that the shooting direction and shooting position of the reference image are the same as those of the target image, it can be determined that the picture content in the reference image and the picture content in the target image have the same perspective and the same scene; based on the fact that the field of view angle of the reference image is larger than the field of view angle of the target image, it can be determined that the field of view range of the picture content in the reference image is larger than the field of view range of the picture content in the target image, thereby, by expanding the picture content in the target image based on the picture content in the reference image that conforms to common sense in reality, a display picture that conforms to common sense in reality can be obtained, thereby improving the accuracy of picture expansion.

[0071] Figure 2 A schematic diagram of the implementation flow of an image processing method provided in an embodiment of the present application. Figure 1 , as described; Figure 1 S102 in can be updated to step S201; Figure 2 The steps shown are explained.

[0072] Step S201, parse the target image to generate a parsing result; the parsing result includes the reference image, and the reference image is the encoded data stored in the file of the target image.

[0073] In the embodiment of the present application, the reference picture is the coded data stored in the target picture; it is understandable that when the reference picture and the target picture are captured by the shooting device, the reference picture is encoded to obtain the coded data of the reference picture, and the coded data of the reference picture is stored in the header file of the target picture. In other embodiments, the reference picture in the picture format can be stored in the header file of the target picture. Among them, compared with storing the reference picture in the picture format in the header file of the target picture, the coded data of the reference picture occupies a smaller space.

[0074] In some embodiments, the encoded data including the reference picture can be obtained in the following manner: parsing the header file of the target picture to obtain a parsing result including the encoded data of the reference picture; parsing the header file of the target picture to obtain a parsing result including the reference picture, encoding the reference picture, and obtaining the encoded data of the reference picture.

[0075] In some embodiments, the encoding method for encoding the reference picture may include: lossless compression, lossy compression, latent space coding, etc., wherein the memory occupied by the encoded data after encoding the reference picture is less than the memory occupied before encoding; illustratively, if the reference picture is encoded by lossless compression, Run-Length Encoding (RLE) and Huffman Coding (Huffman Coding) can be used to compress the reference picture data, and the encoded data is a portable network graphic (PNG); if the reference picture is encoded by latent space coding, the encoded data is a latent space expression. In an embodiment of the present application, the reference picture is encoded by latent space coding to obtain a latent space expression of the reference picture.

[0076] In an embodiment of the present application, the reference picture is stored in the file of the target picture in the format of encoded data to reduce the memory occupied by the reference picture. The encoded data of the reference picture is obtained by parsing the header file of the target picture to improve the data computing efficiency during the expansion of the target picture.

[0077] Figure 3 The following is a schematic diagram of an implementation flow of an image processing method provided in an embodiment of the present application. The analysis result also includes the relative position relationship between the target image and the reference image; based on Figure 1 , Figure 1 Step S103 in can be updated to step S301 to step S305, combining Figure 3 The steps shown are explained.

[0078] Step S301: Reduce the resolution of the target image to obtain a target image with a first resolution.

[0079] In some embodiments, the resolution of the target image may be represented by the number of pixels in the first direction and the number of pixels in the second direction.

[0080] In some embodiments, the number of pixels in the first direction of the target image is reduced from the first number of pixels to the first number of pixels, and the number of pixels in the second direction of the target image is reduced from the third number of pixels to the fourth number of pixels, so as to obtain a target image of a first resolution; wherein the first number of pixels is greater than or equal to the second number of pixels, and the third number of pixels is greater than or equal to the fourth number of pixels. Exemplarily, the resolution of the target image is 1920*1080 as an example to illustrate the embodiments of the present application. The number of pixels in the horizontal direction of the target image is reduced from 1920 to 512; the number of pixels in the vertical direction of the target image is reduced from 1080 to 1080, to obtain a target image with a resolution of 512*1080.

[0081] Step S302: Encode the target image of the first resolution to obtain encoded data to be expanded.

[0082] In some embodiments, the target image of the first resolution is encoded to obtain the encoded data to be expanded; it can be understood that the target image of the first resolution is encoded in the latent space to obtain the latent space encoding to be expanded. Exemplarily, the target image with a resolution of 512*512 is encoded in the latent space to obtain the latent space encoding to be expanded.

[0083] Step S303: Using the target model, based on the relative position relationship between the target picture and the reference picture, feature fusion is performed on the coded data to be expanded and the coded data of the reference picture to obtain target coded data.

[0084] In some embodiments, the relative position relationship between the target image and the reference image represents the relationship between the coordinates of the key points in the target image and the key points in the reference image; exemplarily, the relative position relationship between the target image and the reference image can be the relationship between the coordinates of the person in the target image and the coordinates of the task in the reference image.

[0085] In some embodiments, the target model is used to perform feature fusion on the coded data to be expanded and the coded data of the reference image based on the relative position relationship between the target image and the reference image to obtain target coded data; it can be understood that the target model aligns the coordinates of key points in the target image and the reference image based on the relative position relationship between the target image and the reference image, and performs feature fusion on the coded data of the aligned reference image and the coded data to be expanded to obtain target coded data.

[0086] Exemplarily, the target model aligns the coordinates of information such as people and objects in the target image and the reference image according to the relative position relationship between the target image and the reference image, and performs feature fusion on the latent space expression of the aligned reference image and the latent space expression to be expanded to obtain the target latent space expression.

[0087] Step S304: Decode the target encoded data to obtain a display image of a first resolution.

[0088] In some embodiments, the target coded data is decoded to obtain a display image of a first resolution; it is understandable that the target coded data is latent space decoded to obtain a display image of the first resolution; exemplarily, the target coded data is latent space decoded to obtain a display image with a resolution of 512*512.

[0089] Step S305: enlarge the display image of the first resolution to the target resolution to obtain the display image.

[0090] In some embodiments, the display image of the first resolution is enlarged to the target resolution to obtain the display image; it can be understood that the number of pixels in the horizontal direction of the display image of the first resolution is enlarged to the target number of pixels, and the number of pixels in the vertical direction of the display image of the first resolution is enlarged to the target number of pixels to obtain the display image.

[0091] Exemplarily, a display image with a resolution of 512*512 is super-resolutioned by super-resolution technology to enlarge the pixels of the display image with a resolution of 512*512 to 1920 in the horizontal direction, and enlarge the pixels of the display image with a resolution of 512*512 to 1080 in the vertical direction, so as to obtain a display image with a resolution of 1920*1080.

[0092] In an embodiment of the present application, the resolution of the target image is reduced to a first resolution to reduce memory usage, thereby improving the efficiency of subsequent decoding and encoding; the target model performs feature fusion on the coded data to be expanded and the coded data of the reference image based on the target model according to the relative position relationship between the target image and the reference image to obtain target coded data; a display image of a first resolution is obtained based on the target coded data, and the display image of the first resolution is enlarged to a display image of a target resolution, thereby improving the clarity and accuracy of the display image.

[0093] Figure 4A A schematic diagram of the implementation flow of an image processing method provided in an embodiment of the present application. Figure 3 , the target model includes a first target model and a second target model; Figure 3 Step S303 in can be updated to step S401 and step S402, combining Figure 4A The steps shown are explained.

[0094] Step S401: Using the first target model, perform feature transformation on the relative position relationship between the target image and the reference image and the coding data of the reference image to obtain reference coding data; the reference coding data carries image features of each position in the reference image under the perspective of the target image.

[0095] In some embodiments, it can be understood that the coding data of the reference picture of the relative position relationship between the target picture and the reference picture is input into the first target model, and the target image features of the target person or target object in the reference picture are obtained from the coding data of the reference picture based on the first target, and the reference coding data is generated based on the relationship between the reference picture and the target picture and the target image features; the reference coding data represents the image features existing at various positions in the reference picture from the perspective of the target picture.

[0096] In some embodiments, the first target model may be a decoupled cross-attention model, which is used to separate the cross-attention layers of text features and image features based on a decoupled cross-attention mechanism, so that image prompts and text features can work together. In an embodiment of the present application, the text feature may be the coordinate relationship between the target image and the target object in the reference image; the image feature is the encoded data of the reference image.

[0097] Exemplarily, the coordinate relationship between the target image and the target object in the reference image and the latent space expression of the reference image are input into the decoupled cross-attention model; the decoupled cross-attention model obtains the target image features in the reference image based on the latent space expression of the reference image, and generates the reference latent space expression from the target image features in the reference image based on the coordinate relationship between the target image and the target object or target person in the reference image; wherein the reference latent space expression carries the image features of the target person or target object in the reference image from the perspective of the target image.

[0098] Step S402: using the second target model, based on the reference coded data, perform an iterative k-times image expansion process on the coded data to be expanded, to obtain the target coded data.

[0099] In some embodiments, the second target model may be an iterative model for performing denoising and denoising on data B N times based on features of data A to obtain the target encoded data.

[0100] In some embodiments, through the second target model, based on the characteristics of each position in the reference coded data and the coordinate relationship of the target person or target object, the coded data to be expanded is subjected to the k-times expansion process to obtain the target coded data; it can be understood that based on the characteristics of each position in the reference coded data and the coordinate relationship of the target person or target object, the coded data to be expanded is subjected to the k-times noise addition to noise reduction process to obtain the target coded data.

[0101] In some embodiments, please refer to Figure 4B , Figure 4BA schematic diagram of the implementation flow of a processing method provided in an embodiment of the present application. Figure 4A , Figure 4A The i-th expansion process in step S402 may include steps S4021 to S4023, combining Figure 4B The steps shown are explained.

[0102] Step S4021, use the reference coding data to perform image information generation processing on the i-th input coding data to obtain the i-th generated overall expression; i, k are positive integers, i is less than or equal to k.

[0103] In some embodiments, during the first image expansion process, the coded data to be expanded is denoised by 100%, and the coded data to be expanded with 100% noise is input into the second target model for image information processing to obtain the first generated overall expression, wherein x is a real number greater than 0 and less than 100. During the second image expansion process, the first generated overall expression is denoised for a second time to obtain the first generated overall expression with y% noise, which is used as the input of the second target model for image information processing to obtain the second generated overall expression, wherein y is a real number less than x and greater than 0.

[0104] Step S4022: Perform the i-th noise addition process on the coded data to be expanded to obtain the i-th original overall expression.

[0105] In some embodiments, the image expansion process in step S4021 is repeated i times to obtain an i-th overall expression that meets preset requirements.

[0106] Step S4023, based on the mask image, the i-th generated overall expression and the i-th original overall expression are merged to obtain the i-th output coded data; the mask image carries the relative position relationship between the target image and the display image; wherein the first input coded data is obtained by performing the first noise addition process on the coded data of the target image; the i-th output coded data is the i+1-th input coded data; the target coded data is the k-th output coded data.

[0107] In some embodiments, the mask image is obtained by the first target model based on the relative position relationship between the reference image and the target image and the encoded data of the reference image; it can be understood that the mask image can be used as the content to be expanded in the target image; corresponding to the above-mentioned encoded data to be expanded.

[0108] In some embodiments, the i-th generated overall expression and the i-th original overall expression are fused based on a mask image to obtain the i-th output encoded data; it can be understood that the i-th original overall expression is used as a background image, the i-th generated overall expression is used as a foreground image, the background image and the foreground image are fused based on the mask image to obtain the i-th output encoded data, and the i-th output encoded data is used as the target encoded data.

[0109] In an embodiment of the present application, reference coding data is obtained through a first target model, and the second model performs k expansion processes on the coding data to be expanded based on the reference coding data to obtain target coding data, thereby improving the accuracy of expansion.

[0110] Figure 5 A schematic diagram of the implementation flow of an image processing method provided in an embodiment of the present application. Figure 3 , Figure 3 Step S302 in can be updated to step S501 and step S502, combining Figure 5 The steps shown are explained.

[0111] Step S501: Encode the target image of the first resolution to obtain encoded data of the target image.

[0112] In some embodiments, latent space encoding is performed on the target image of the first resolution to obtain a latent space encoding of the target image; exemplarily, latent space encoding is performed on the target image with a resolution of 512*512 to obtain a latent space expression of the target image with a resolution of 512*512.

[0113] Step S502: based on the expansion requirement information corresponding to the display image, resize the coded data of the target image to obtain the coded data to be expanded; the coded data to be expanded corresponds to the size of the target image.

[0114] In some embodiments, the extended requirement information corresponding to the displayed image represents extended requirement information for the target image.

[0115] In some embodiments, it can be understood that, based on the expansion requirement information of the target image, the latent space expression of the target image is resized to obtain the latent space expression to be expanded. Wherein, the latent space expression of the target image is resized, and it can be understood that the latent space expression of the target image is reduced to achieve a size corresponding to the latent space expression of the reference image.

[0116] In an embodiment of the present application, based on the expansion requirement information corresponding to the display image, the latent space expression of the target image is resized to improve the accuracy of expanding the target image.

[0117] Figure 6 A schematic diagram of the implementation flow of an image processing method provided in an embodiment of the present application. Figure 2 The analysis result also includes the relative position relationship between the target image and the reference image; the process also includes steps S601 to S603, combining Figure 6 The steps shown are explained.

[0118] Step S601: Decode the encoded data of the reference picture to obtain a compressed reference picture.

[0119] In some embodiments, it can be understood that the latent space expression of the reference picture is decoded to obtain a compressed reference picture.

[0120] Step S602: Based on the extended requirement information corresponding to the display picture and the relative position relationship between the target picture and the reference picture, the compressed reference picture is aligned and cropped to obtain a compressed display picture.

[0121] In some embodiments, it can be understood that based on the extended requirement information of the target image and the coordinate relationship between the target image and the target object or target object in the reference image, the compressed reference image is aligned with the target image, and the compressed reference image is cropped to obtain a compressed display image.

[0122] Step S603: enlarge the resolution of the compressed display image to the target resolution to obtain the display image.

[0123] In some embodiments, it is understandable that the resolution of the compressed display image is enlarged to the target resolution based on the super-resolution technology to obtain the display image.

[0124] Exemplarily, the super-resolution technology is used to perform super-resolution processing on the display image with a compressed resolution of 512*512, so as to enlarge the pixels in the horizontal direction of the display image with a compressed resolution of 512*512 to 1920, and enlarge the pixels in the vertical direction of the display image with a compressed resolution of 512*512 to 1080, so as to obtain a display image with a resolution of 1920*1080.

[0125] In some embodiments, the step of enlarging the resolution of the compressed display picture to the target resolution to obtain the display picture includes: enlarging the resolution of the compressed display picture to the target resolution to obtain the picture to be fused; and fusing the picture to be fused and the target picture based on the extended requirement information corresponding to the display picture and the relative position relationship between the target picture and the reference picture to obtain the display picture.

[0126] In an embodiment of the present application, a compressed reference picture is obtained by decoding the latent space expression of a reference picture, and the compressed reference picture is cropped based on the expansion requirements of the target picture and the relative position relationship between the target picture and the reference picture to obtain a compressed display picture; the compressed display picture is super-resolution processed to obtain the display picture to improve the expansion accuracy of the target picture.

[0127] Figure 7 A schematic diagram of the implementation flow of an image processing method provided in an embodiment of the present application. Figure 2 The analysis result also includes the relative position relationship between the target image and the reference image; the process also includes steps S701 to S607, combining Figure 7 The steps shown are explained.

[0128] Step S701: Decode the encoded data of the reference picture to obtain a compressed reference picture.

[0129] In some embodiments, it can be understood that the latent space expression of the reference picture is decoded to obtain a compressed reference picture.

[0130] Step S702: enlarge the resolution of the compressed reference picture to a target resolution to obtain a target reference picture.

[0131] In some embodiments, it can be understood that the resolution of the compressed reference image is enlarged to the target resolution based on the super-resolution technology to obtain the target reference image; exemplarily, the reference image with a compressed resolution of 512*512 is super-resolution processed by the super-resolution technology to enlarge the horizontal pixels of the reference image with a compressed resolution of 512*512 to 1920, and the vertical pixels of the reference image with a compressed resolution of 512*512 are enlarged to 1080, so as to obtain a target reference image with a resolution of 1920*1080.

[0132] Step S703: Based on the extended requirement information corresponding to the display picture and the relative position relationship between the target picture and the reference picture, image alignment, cropping and image fusion processing are performed on the target reference picture and the target picture to generate the display picture.

[0133] In some embodiments, the expansion requirement information corresponding to the displayed image, that is, the expansion requirement information of the target image, may be to expand the left side of the target image; or to expand the four directions of the target image by 30% as a whole.

[0134] In some embodiments, it can be understood that based on the extended requirement information of the target image and the coordinate relationship between the target image and the target object or target object in the reference image, the target reference image is aligned with the target image, and the aligned target reference image is cropped to obtain a cropped target reference image; the cropped target reference image is fused with the target image to obtain the display image.

[0135] In an embodiment of the present application, the resolution of the compressed reference image is enlarged to obtain a high-resolution reference image; based on the extended requirement information corresponding to the display image and the relative position relationship of the coordinates, the high-resolution reference image is aligned, cropped, and fused to obtain a display image with high accuracy and resolution.

[0136] Figure 8 The method may further include step S801 and step S802. Figure 8 The steps shown are explained.

[0137] Step S801: In response to obtaining the compressed reference picture, analyzing the compressed reference picture.

[0138] In some embodiments, it can be understood that in response to decoding the latent space expression of the reference picture and obtaining a compressed reference picture, the target content in the compressed reference picture is analyzed; wherein, the target content in the compressed reference picture is analyzed, including: obtaining the target person or target object in the compressed reference picture, based on the extended requirements corresponding to the display picture, analyzing whether the target person or target object in the compressed reference picture meets the extended requirements, and generating an analysis result.

[0139] In some embodiments, if the analysis result includes that the target person or target object in the compressed reference image meets the expansion requirements, the compressed reference image is aligned, cropped, and fused to obtain the display image.

[0140] Step S802: When the analysis result indicates that the compressed reference picture contains target content that does not meet the expansion requirement, modify the target content in the compressed reference picture.

[0141] In some embodiments, it can be understood that when the target person or target object in the compressed reference picture does not meet the expansion requirements corresponding to the display picture, based on the expansion requirements of the target picture, the target person or target object in the reference picture is modified to obtain a reference picture that can meet the expansion requirements of the target picture.

[0142] In an embodiment of the present application, based on the analysis of the target content in the reference picture, it is determined whether the target content in the reference picture meets the expansion requirements of the target picture. If not, the target content in the reference picture is modified to improve the accuracy of the expansion of the target picture.

[0143] Fig. 9 The method may further include steps S901 to S903, which are combined with Fig. 9 The steps shown are explained.

[0144] Step S901, in response to a photo taking instruction, obtaining a target image and a corresponding reference image including the same object; the shooting position of the target image is the same as the shooting position of the reference image, the shooting direction of the target image is the same as the shooting direction of the reference image, and the field of view angle of the target image is smaller than the field of view angle of the reference image.

[0145] In some embodiments, it can be understood that in response to a shooting instruction for a target person or target object, the target person or target object is shot by at least one shooting device at the same position and in the same direction to obtain a target image and a reference image, wherein the field of view angle of the target image is smaller than the field of view angle of the reference image.

[0146] In some embodiments, when the target person or target object is photographed by a photographing device to obtain a target picture and a reference picture, it includes: when the target person or target object is photographed by the photographing device at the same time, the target picture is obtained by photographing the wide-angle lens of the photographing device, and the reference picture is obtained by photographing the wide-angle lens of the photographing device. When the target person or target object is photographed by the photographing device at different times, the shooting position and shooting direction of the target picture and the reference picture are the same, and the field of view angle of the photographing device when shooting the reference picture is greater than the field of view angle of the photographing device when shooting the target picture; it can be understood that the field of view of the target person or target object in the reference picture is greater than the field of view of the target person or target object in the target picture.

[0147] Step S902: Save the reference image and the target image.

[0148] In some embodiments, it is understandable that the reference image and the target image are saved in the memory of the shooting device, wherein the target image and the reference image are associated with each other, and when responding to an instruction to expand the target image, the reference image is called.

[0149] In other embodiments, it can be understood that the reference image is saved in the header file of the target image, and when responding to an instruction to expand the target image, the header file of the target image is parsed to obtain the reference image; or, the latent space expression of the reference image is saved in the header file of the target image, and when responding to an instruction to expand the target image, the header file of the target image is parsed to obtain the latent space expression of the reference image.

[0150] Step S903, display the target image, the reference image is used to respond to the expansion instruction for the target image, and generate a display image together with the target image based on the target model, the size of the display image is larger than the size of the target image, and the display image includes the image content of the target image and the expanded content.

[0151] In some embodiments, it is understandable that the target image is displayed through a display screen of a shooting device; in response to an extension instruction for the target image, the reference image is called according to the associated information between the target image and the reference image; or the header file of the target image is parsed to obtain the reference image or a latent space expression of the reference image; based on the reference image, extended content for the target image is generated, and a display image is generated based on a target model using the extended content and the target image, wherein the size of the display image is larger than the size of the target image; it is understandable that the extended display image needs to be super-resolution processed to obtain a high-resolution display image.

[0152] In an embodiment of the present application, in response to an expansion instruction for the target image, the target model generates expanded content for the target image based on a reference image, and generates the display image based on the expanded content and the target image, thereby improving the accuracy of image expansion.

[0153] Fig.10 A schematic diagram of the implementation flow of an image processing method provided in an embodiment of the present application. Fig. 9 , Fig. 9 Step S902 in can be updated to step S1001 or step S1002, which will be combined Fig.10 The steps shown are explained.

[0154] Step S1001: Save the reference image to the header file of the target image.

[0155] In some embodiments, it can be understood that the reference picture in the Joint Photographic Experts Group (JPEG) format is saved in the header file of the target picture based on the JPEG multi-image file format.

[0156] Step S1002: encode the reference picture to obtain the encoded data of the reference picture; and save the encoded data of the reference picture to the header file of the target picture.

[0157] In some embodiments, the reference picture is subjected to latent space coding to obtain a latent space coding of the reference picture, and the latent space expression of the reference picture is saved in a header file of the target picture.

[0158] In some other embodiments, the index information of the reference picture may be stored in a header file of the target picture, and the reference picture or the latent space expression of the reference picture may be stored in the tail of the target picture.

[0159] In an embodiment of the present application, the reference picture or the latent space expression of the reference picture is stored in the header file of the target picture to improve the efficiency of expanding the target picture.

[0160] Fig.11 A schematic diagram of the implementation flow of an image processing method provided in an embodiment of the present application. Fig.10 , Fig.10 Step S1002 in can be updated to step S1101 to step S1103, combining Fig.11 The steps shown are explained.

[0161] Step S1101: Perform distortion correction processing on the reference image to obtain a corrected reference image.

[0162] In some embodiments, it is understandable that the reference image is subjected to distortion correction processing to eliminate the influence of image deformation caused by the lens of the shooting device or the imaging process, so as to obtain a corrected reference image.

[0163] Step S1102: Downsize the corrected reference image to obtain a compressed reference image.

[0164] In some embodiments, it is understandable that the size of the corrected reference picture is reduced, that is, the resolution of the corrected reference picture is reduced to obtain a compressed reference picture.

[0165] Step S1103: Encode the compressed reference picture to obtain encoded data of the reference picture.

[0166] In some embodiments, it can be understood that latent space encoding is performed on the compressed reference picture to obtain a latent space expression of the reference picture.

[0167] In an embodiment of the present application, distortion correction processing is performed on the reference image to eliminate the influence of image deformation caused by the lens of the shooting device or the imaging process, so as to improve the accuracy of expanding the target image.

[0168] An exemplary application of an image processing method provided in an embodiment of the present application in a practical scenario is described below.

[0169] With the advancement of technology, AI image expansion has become a popular feature in the industry recently. AI image expansion is based on the original real shot image and uses artificial intelligence generated content (AIGC) technology to expand the viewing angle of the target image. Fig.12 As shown, Fig.121201 is an original image, 1202 is an image after the original image is expanded 1.5 times by the AIGC technology, and 1203 is an image after the original image is expanded 2 times by the AIGC technology. In particular, the image expansion by the AIGC technology still violates the reality, such as Fig.12 In 1203, the extended area below the character lacks a rope, which is obviously in conflict with common sense in reality.

[0170] For the above technical issues, please refer to Fig.13 , Fig.13 A schematic diagram of the implementation flow of a method for expanding a map provided in an embodiment of the present application. The method includes steps S1301 to S1303. Fig.13 The steps shown are explained.

[0171] Step S1301: embed the distortion-corrected miniature wide-angle image of the same scene into the head of the main camera image or the telephoto image.

[0172] In some embodiments, when a target object is photographed through a main camera lens or a telephoto lens of a camera or other mobile device to obtain a main camera picture or a telephoto picture, a wide-angle picture of the target object is simultaneously photographed through a wide-angle lens of a camera or other mobile device to obtain a miniature wide-angle picture of the same scene that has been corrected for distortion is embedded in the header of the main camera picture or the telephoto picture; it can be understood that the miniature wide-angle picture that has been corrected for distortion is embedded in the header file of the main camera picture or the telephoto picture.

[0173] In some embodiments, after the wide-angle image is deformed, the key points in the wide-angle image are aligned with the key points in the main camera image or the telephoto image, and a mapping relationship between the coordinates in the main camera image or the telephoto image and the wide-angle image is generated. The wide-angle image is reduced based on the mapping relationship between the coordinates in the main camera image or the telephoto image and the wide-angle image to reduce the impact of the large size of the wide-angle image on the size of the main camera image or the telephoto image, and the coordinate correspondence between the reduced-size wide-angle image and the main camera image or the telephoto image is stored in the header file of the main camera image or the telephoto image; wherein, the index information of the reduced-size wide-angle image can also be stored in the header file of the main camera image or the telephoto image, and the coordinate correspondence between the reduced wide-angle image and the main camera image or the telephoto image is stored in the tail file of the main camera image or the telephoto image.

[0174] In some embodiments, please refer to Fig.14 , Fig.14 The following is a schematic diagram of a wide-angle image storage implementation provided in an embodiment of the present application. Fig.14As shown, 1401 represents a wide-angle image after distortion correction; 1402 represents reducing the wide-angle image after distortion correction to obtain a reduced wide-angle image; 1403 represents writing the reduced wide-angle image into the head extended data area of ​​the main camera image or the telephoto image; 1404 represents saving the main camera image or the telephoto image and its head extended data area.

[0175] Step S1302: when the main camera image or the telephoto image needs to be expanded, parse the wide-angle image from the header file of the main camera image or the telephoto image.

[0176] Step S1303: Based on the main camera picture or the telephoto picture and the wide-angle picture, the main camera picture or the telephoto picture is expanded by using AI super-resolution technology and AIGC technology to obtain an expanded main camera picture or the telephoto picture.

[0177] In some embodiments, AI super-resolution technology is an image processing technology based on deep learning and neural network algorithms, which aims to upgrade low-resolution images or videos to high-resolution; it can be understood that AI super-resolution technology is used to obtain materials from low-pixel pictures or videos to generate high-resolution pictures; among them, the current AI super-resolution technology supports 8x super-resolution processing. For example, a 320*240 wide-angle thumbnail is super-resolutioned 8 times, and the resolution can reach 2560*1920.

[0178] In some embodiments, the wide-angle picture obtained from the main camera picture or the telephoto picture is enlarged to the target resolution using AI super-resolution technology, and then aligned based on the coordinate correspondence and the key points of the main camera picture or the wide-angle picture. The aligned wide-angle picture and the main camera picture or the wide-angle picture are fused and transitioned so that the wide-angle picture and the main camera picture or the wide-angle picture are fused into one picture.

[0179] In some embodiments, when the AI ​​model is used to expand the main camera image or the telephoto image, it is based on the miniature wide-angle image as a reference, and does not rely entirely on the information of the miniature wide-angle image; it is understandable that if the wide-angle image contains content that is not suitable for display, the AI ​​model can be prompted through voice commands or text commands to ignore the content that is not suitable for display; among them, content that is not suitable for display may be garbage, violent scenes, etc.

[0180] In other embodiments, AIGC technology can also be used to take the main camera image or telephoto image and the miniature wide-angle image as input, design and train the model, and generate a highly realistic extended image based on the trained model.

[0181] Fig.15 A schematic diagram of an implementation method of an image expansion method provided in an embodiment of the present application is shown in FIG. Fig.15As shown, 1501 indicates reading a wide-angle picture from the head extension area of ​​the main camera picture or the telephoto picture; 1502 indicates using the AI ​​super-resolution model to enlarge the resolution of the wide-angle picture; 1503 indicates that the model based on AIGC technology fuses the main camera picture or the telephoto picture with the enlarged wide-angle picture to obtain an extended picture.

[0182] Fig.16A A schematic diagram of an implementation principle of a shooting principle is provided for an embodiment of the present application; please refer to Figure 16, wherein 1601 represents a main camera, 1602 represents a telephoto camera, and 1603 represents a wide-angle camera; when a user shoots with the main camera, the main camera 1601 is used to respond to a shooting instruction and obtain a main camera image of the current scene; the wide-angle camera 1603 is used to obtain a wide-angle image, and after performing distortion correction on the wide-angle image, align the wide-angle image based on the main camera image, and generate corresponding point information; the wide-angle image is reduced; the main camera 1601 is also used to store the reduced wide-angle image and the corresponding point information in the header file of the main camera image, and encode the main camera image, the reduced wide-angle image and the corresponding point information to obtain the encoded main camera image.

[0183] Fig. 16B A schematic diagram of the implementation of a shooting principle is provided for the embodiment of the present application; Fig. 16B As shown, 1610 represents a telephoto camera, 1620 represents a main camera, and 1630 represents a wide-angle camera; when a user takes a photo through the telephoto camera 1610, the telephoto camera 1610 is used to respond to a shooting instruction and obtain a telephoto picture of the current scene; the wide-angle camera 1630 is used to obtain a wide-angle picture, and after performing distortion correction on the wide-angle picture, align the wide-angle picture based on the telephoto picture, and generate corresponding point information; the wide-angle picture is reduced; the telephoto camera 1610 is also used to store the reduced wide-angle picture and the corresponding point information in the header file of the telephoto picture, and encode the telephoto picture, the reduced wide-angle picture and the corresponding point information to obtain the encoded telephoto picture.

[0184] Fig.17 A schematic diagram of the implementation flow of a method for expanding a map is provided for an embodiment of the present application. The method includes steps S1701 to S1707. Figure 7 The steps shown are explained.

[0185] Step S1701, expanding the viewing angle of the selected image.

[0186] In some embodiments, in the selected pictures to be expanded, the viewing angle to be expanded is selected to obtain the viewing angle picture to be expanded.

[0187] Step S1702: parse the header file of the image to be extended.

[0188] In some embodiments, a header file of the image to be expanded is parsed.

[0189] Step S1703: Check whether the header file of the image to be extended includes a miniature wide-angle sub-image.

[0190] Step S1704: Use the miniature wide-angle sub-image and the image to be expanded as input to the image expansion model.

[0191] Step S1705: Based on the miniature wide-angle sub-image and the image to be expanded, the image is expanded by using a customized AIGC image expansion model to obtain an expanded image.

[0192] Step S1706: Use the image to be expanded as the input of the image expansion model.

[0193] Step S1707: Based on the view picture to be expanded, expand the picture by using a common AIGC picture expansion model to obtain an expanded picture.

[0194] Fig.18A A schematic diagram of an implementation of a shooting end architecture provided in an embodiment of the present application is shown as follows: Fig.18A As shown, it includes the architecture 1801 of the main camera, the architecture 1802 of the wide-angle camera, and the encoder 1803; wherein, the architecture 1801 of the main camera is used to send the main camera picture in RAW format to the encoder 1803 after passing through the RGB domain and the YUV domain; the architecture 1802 of the wide-angle camera is used to take pictures of the same scene with the wide angle when taking pictures through the main camera (or telephoto); the wide-angle picture in RAW format taken by the wide-angle camera is directly converted into a latent space expression in the RGB domain through the latent space encoder, and the latent space expression of the wide-angle picture is sent to the encoder N03, and is also used to obtain the corresponding relationship between the coordinates in the main camera picture and the wide-angle picture according to the main camera picture in the RGB domain, and store the corresponding relationship of the coordinates in the header file of the main camera picture; the encoder 1803 encodes the latent space expression of the main camera picture and the wide-angle picture with the corresponding relationship of the coordinates stored, and obtains the encoded main camera picture. Among them, the encoded main camera picture can be used as the input of the AIGC expansion operation on the album side. It is understandable that storing the latent space expression in the header file of the main camera image can not only reduce the space occupied (the latent space encoder has a higher compression rate than JPEG) but also speed up the subsequent AIGC image expansion operation compared to storing JPG.

[0195] Fig.18B A schematic diagram of implementing an extended image provided in an embodiment of the present application is shown in FIG. Fig.18BAs shown, based on the demand for expanding the main camera image in multiple directions, the main camera image is expanded based on the wide-angle image to obtain an extended image 1810. According to the extended image 1810, compared with the target image, it is expanded by 64% to the left, 36% to the right, 48% upward, and 52% downward.

[0196] In some embodiments, when the main camera (or telephoto) image is expanded, the latent space expression of the wide-angle image is directly used to restore the wide-angle image. Then, according to the positional relationship between the main camera (or telephoto) image and the wide-angle image (the perspective positional relationship determined during the shooting stage and stored in EXIF), as well as the direction and proportion of the user's expansion, it is determined whether to crop or further expand the restored wide-angle image without reference.

[0197] Fig.19 The following is a schematic diagram of an implementation of a method for expanding a map provided in an embodiment of the present application. Fig.19 As shown, it includes a main camera picture 1901, a noise adding module 1902, a decoding module 1903, a wide-angle picture 1904, a cropping module 1905, a cropped wide-angle picture 1906, a super-resolution module 1907 and an extended picture 1908; wherein, the main camera picture 1901 is parsed to obtain a latent space expression of the wide-angle picture and a positional relationship between the wide-angle picture and the main camera picture; the latent space expression of the wide-angle picture is denoised by the noise adding module 1902; the latent space expression after noise is latently decoded by the decoding module 1903 to obtain the wide-angle picture 1904; the wide-angle picture 1904 is cropped by the cropping module 1905 to obtain a cropped wide-angle picture 1906; the cropped wide-angle picture 1906 is super-resolution processed by the super-resolution module 1907 to obtain an extended picture 1908.

[0198] Fig. 20 The following is a schematic diagram of an implementation of a method for expanding a map provided in an embodiment of the present application. Fig. 20As shown, it includes a main camera picture 2001, a reduced main camera picture 2002, an encoding module 2003, a noise adding module 2004, a parsing module 2005, a first target model 2006, a second target model 2007, a target latent space expression 2008, a decoding module 2009, an extended picture 2010, a super-resolution module 2011, and a target extended picture 2012; wherein, the size of the main camera picture 2001 is reduced to obtain the reduced main camera picture 2002; the encoding module 2003 performs latent space decoding on the reduced main camera picture 2002 to obtain the latent space expression of the reduced main camera picture 2002; the noise adding module 2004 performs the decompression on the reduced main camera picture The latent space expression of 2002 is denoised to obtain the latent space expression of the main camera picture after the noise is added; the main camera picture 2001 is processed by the parsing module 2005; the reference latent space expression is obtained based on the latent space expression of the wide-angle picture and the positional relationship between the wide-angle picture and the main camera picture by the first model 2006; the target latent space expression 2008 is obtained based on the latent space expression of the main camera picture after the noise is added and the reference latent space expression by the second model 2007; the target latent space expression 2008 is latently decoded by the decoding module 2009 to obtain the extended picture 2010; the extended picture 2010 is super-resolved by the super-resolution module 2011 to obtain the target extended picture 2012.

[0199] In an embodiment of the present application, the main camera picture or the telephoto picture is expanded based on the target model to obtain an extended picture through the latent space expression of the wide-angle picture or the wide-angle picture stored in the header file of the main camera picture or the telephoto picture, and the relative position relationship between the main camera picture or the telephoto picture and the wide-angle picture, and the extended picture is processed based on the super-resolution technology to obtain an extended picture of the target resolution; the accuracy of picture expansion and the resolution of the extended picture are improved.

[0200] Based on the foregoing embodiments, an embodiment of the present application provides an image processing device, which includes the units included and the modules included in the units, which can be implemented by a processor in an electronic device; of course, it can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP) or a field programmable gate array (FPGA), etc.

[0201] Fig.21 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application is shown in FIG. Fig.21As shown, the image processing device 2100 includes: a first acquisition module 2101, a second acquisition module 2102 and a generation module 2103; wherein: the first acquisition module 2101 is used to obtain a target image; the second acquisition module 2102 is used to obtain a reference image; the shooting position of the target image is the same as the shooting position of the reference image, the shooting direction of the target image is the same as the shooting direction of the reference image, and the field of view angle of the target image is smaller than the field of view angle of the reference image; the generation module 2103 is used to generate a display image based on a target model, a reference image and a target image; wherein the size of the display image is larger than the size of the target image, the display image includes the image content of the target image and extended content, and the extended content is generated based on the target model, the reference image and the target image to expand the display content in at least one direction of the target image.

[0202] In some embodiments, the second obtaining module 2102 is further used to parse the target image and generate a parsing result; the parsing result includes the reference image, and the reference image is the encoded data stored in the file of the target image.

[0203] In some embodiments, the parsing result also includes the relative position relationship between the target image and the reference image; the generation module 2103 is also used to reduce the resolution of the target image to obtain a target image of a first resolution; encode the target image of the first resolution to obtain the encoded data to be expanded; through the target model, based on the relative position relationship between the target image and the reference image, feature fusion is performed on the encoded data to be expanded and the encoded data of the reference image to obtain target encoded data; the target encoded data is decoded to obtain a display image of a first resolution; the display image of the first resolution is enlarged to the target resolution to obtain the display image.

[0204] In some embodiments, the target model includes a first target model and a second target model; the generating module 2103 is further used to perform feature transformation on the relative position relationship between the target image and the reference image and the coding data of the reference image through the first target model to obtain reference coding data; the reference coding data carries the image features of each position in the reference image under the viewing angle of the target image; and the coding data to be expanded is iterated k times based on the reference coding data through the second target model to obtain the target coding data;

[0205] The i-th image expansion process includes:

[0206] Using the reference coded data, image information generation processing is performed on the ith input coded data to obtain the ith generated overall expression; i, k are positive integers, i is less than or equal to k;

[0207] Performing an i-th noise addition process on the coded data to be expanded to obtain an i-th original overall expression;

[0208] fusing the i-th generated overall expression and the i-th original overall expression based on the mask image to obtain i-th output encoded data; the mask image carries the relative position relationship between the target image and the display image;

[0209] Among them, the first input encoded data is obtained by performing the first noise addition process on the encoded data of the target image; the i-th output encoded data is the i+1-th input encoded data; and the target encoded data is the k-th output encoded data.

[0210] In some embodiments, the image processing device 2100 further includes an encoding module (not shown in the figure), which is used to perform encoding processing on the target image of the first resolution to obtain encoded data of the target image;

[0211] Based on the expansion requirement information corresponding to the display image, the encoding data of the target image is resized to obtain the encoding data to be expanded; the encoding data to be expanded corresponds to the size of the target image.

[0212] In some embodiments, the parsing result also includes the relative position relationship between the target image and the reference image; the generation module 2103 is also used to decode the encoded data of the reference image to obtain a compressed reference image; based on the extended requirement information corresponding to the display image and the relative position relationship between the target image and the reference image, the compressed reference image is aligned and cropped to obtain a compressed display image; the resolution of the compressed display image is enlarged to the target resolution to obtain the display image.

[0213] In some embodiments, the generation module 2103 is also used to decode the encoded data of the reference picture to obtain a compressed reference picture; enlarge the resolution of the compressed reference picture to the target resolution to obtain a target reference picture; and perform image alignment and cropping and image fusion processing on the target reference picture and the target picture based on the extended requirement information corresponding to the display picture and the relative position relationship between the target picture and the reference picture to generate the display picture.

[0214] In some embodiments, the processing device 2100 further includes an analysis module (not shown in the figure), configured to analyze the compressed reference picture in response to obtaining the compressed reference picture;

[0215] When the analysis result indicates that the compressed reference picture contains target content that does not meet the expansion requirement, the target content in the compressed reference picture is modified.

[0216] Fig. 22 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application is shown in FIG. Fig. 22 As shown, the image processing device 2200 includes: a third acquisition module 2201, a storage module 2202 and a display module 2203, wherein: the third acquisition module 2201 is used to respond to a photo taking instruction to obtain a target image and a corresponding reference image including the same object; the shooting position of the target image is the same as the shooting position of the reference image, the shooting direction of the target image is the same as the shooting direction of the reference image, and the field of view angle of the target image is smaller than the field of view angle of the reference image; the storage module 2202 is used to save the reference image and the target image; the display module 2203 is used to display the target image; the reference image is used to respond to an extension instruction for the target image, and generate a display image together with the target image based on a target model, the size of the display image is larger than the size of the target image, and the display image includes the image content of the target image and the extended content.

[0217] In some embodiments, the saving module 2202 is also used to save the reference image to the header file of the target image; or, encode the reference image to obtain the encoded data of the reference image; and save the encoded data of the reference image to the header file of the target image.

[0218] In some embodiments, the image processing device 2200 also includes an encoding module (not shown in the figure), which is used to perform distortion correction processing on the reference image to obtain a corrected reference sub-image; reduce the corrected reference sub-image to obtain a compressed reference image; and encode the compressed reference image to obtain encoded data of the reference image.

[0219] The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. In some embodiments, the functions or modules included in the device provided in the embodiment of the present application can be used to execute the method described in the above method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0220] It should be noted that in the embodiment of the present application, if the above method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium, including a number of instructions to enable an electronic device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific hardware, software or firmware, or any combination of hardware, software, and firmware.

[0221] An embodiment of the present application provides an electronic device, including a display screen, a memory and a processor. The display screen is used to display pictures. The memory stores a computer program that can be run on the processor. When the processor executes the program, some or all of the steps in the above method are implemented.

[0222] The embodiment of the present application provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, some or all of the steps in the above method are implemented. The computer-readable storage medium can be transient or non-transient.

[0223] An embodiment of the present application provides a computer program, including a computer-readable code. When the computer-readable code is run in an electronic device, a processor in the electronic device executes some or all of the steps for implementing the above method.

[0224] The embodiment of the present application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and when the computer program is read and executed by a computer, some or all of the steps in the above method are implemented. The computer program product can be implemented specifically by hardware, software or a combination thereof. In some embodiments, the computer program product is specifically embodied as a computer storage medium, and in other embodiments, the computer program product is specifically embodied as a software product, such as a software development kit (SDK) and the like.

[0225] It should be noted here that the description of the various embodiments above tends to emphasize the differences between the various embodiments, and the same or similar aspects can be referenced to each other. The description of the above device, storage medium, computer program and computer program product embodiments is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the embodiments of the device, storage medium, computer program and computer program product of this application, please refer to the description of the method embodiment of this application for understanding.

[0226] Fig.23 A hardware entity diagram of an electronic device provided in an embodiment of the present application, such as Fig.23 As shown, the hardware entity of the electronic device 2300 includes: a processor 2301 and a memory 2302, wherein the memory 2302 stores a computer program that can be run on the processor 2301, and the processor 2301 implements the steps in the method of any of the above embodiments when executing the program.

[0227] The memory 2302 stores computer programs that can be run on the processor. The memory 2302 is configured to store instructions and applications executable by the processor 2301. It can also cache data to be processed or processed by the processor 2301 and various modules in the electronic device 2300 (for example, image data, audio data, voice communication data, and video communication data). This can be achieved through flash memory (FLASH) or random access memory (Random Access Memory, RAM).

[0228] When the processor 2301 executes the program, the steps of any of the above methods are implemented. The processor 2301 generally controls the overall operation of the electronic device 2300.

[0229] Fig.24 A hardware entity diagram of an electronic device provided in an embodiment of the present application, such as Fig.24As shown, the hardware entity of the electronic device 2400 includes: a display screen 2401 and an image processing device 2402, wherein the display screen 2401 is used to display pictures; the image processing device 2402 includes a first acquisition component 2403, a second acquisition component 2404 and a generation component 2405; wherein the first acquisition component 2403 is used to obtain a target picture; the second acquisition component 2404 is used to obtain a reference picture, the shooting position of the target picture is the same as the shooting position of the reference picture, the shooting direction of the target picture is the same as the shooting direction of the reference picture, and the field of view angle of the target picture is smaller than the field of view angle of the reference picture; the generation component 2405 is used to generate a display picture based on the target model, the reference picture and the target picture; wherein the size of the display picture is larger than the size of the target picture, and the display picture includes the picture content of the target picture and the extended content, and the extended content is the display content generated based on the target model, the reference picture and the target picture for expanding at least one direction of the target picture; the display screen 2401 is used to display the target picture.

[0230] An embodiment of the present application provides a computer storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement the steps of the method of any of the above embodiments.

[0231] It should be noted here that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0232] The processor may be at least one of an Application Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), a Digital Signal Processing Device (DSPD), a Programmable Logic Device (PLD), a Field Programmable Gate Array (FPGA), a Central Processing Unit (CPU), a controller, a microcontroller, and a microprocessor. It is understandable that the electronic device that implements the functions of the processor may also be other, and the embodiments of the present application are not specifically limited.

[0233] The above-mentioned computer storage medium / memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory (Flash Memory), a magnetic surface memory, an optical disk, or a compact disc read-only memory (CD-ROM) and the like; it can also be various terminals including one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc.

[0234] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the serial number of each step / process mentioned above does not mean the order of execution, and the execution order of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application. The serial numbers of the embodiments of the present application mentioned above are for description only and do not represent the advantages and disadvantages of the embodiments.

[0235] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0236] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0237] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0238] In addition, all functional units in the embodiments of the present application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the above integrated unit can be implemented in the form of hardware or in the form of hardware plus software functional units. A person of ordinary skill in the art can understand that all or part of the steps of the above method embodiments can be completed by hardware related to program instructions, and the above program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above method embodiments are executed; and the above storage medium includes: mobile storage devices, read-only memory (ROM), disks or optical disks, etc. Various media that can store program codes.

[0239] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for an electronic device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0240] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. An image processing method, comprising: Get the target image; Obtaining a reference picture, wherein the shooting position of the target picture is the same as that of the reference picture, the shooting direction of the target picture is the same as that of the reference picture, and the field of view angle of the target picture is smaller than that of the reference picture; Generate a display picture based on the target model, the reference picture and the target picture; Among them, the size of the display picture is larger than the size of the target picture, and the display picture includes the picture content of the target picture and extended content, and the extended content is generated based on the target model, the reference picture and the target picture to expand the display content of at least one direction of the target picture.

2. The method according to claim 1, wherein obtaining a reference picture comprises: Parsing the target image to generate a parsing result; The parsing result includes the reference picture, which is encoded data stored in a file of the target picture.

3. The method according to claim 2, wherein: The parsing result also includes a relative position relationship between the target image and the reference image; The generating of the display picture based on the target model, the reference picture and the target picture includes: Reducing the resolution of the target image to obtain a target image of a first resolution; Performing encoding processing on the target image of the first resolution to obtain encoded data to be expanded; By using the target model, based on the relative position relationship between the target picture and the reference picture, feature fusion is performed on the coded data to be extended and the coded data of the reference picture to obtain target coded data; Decoding the target coded data to obtain a display picture of a first resolution; The display image of the first resolution is enlarged to the target resolution to obtain the display image.

4. The method according to claim 3, wherein the target model comprises a first target model and a second target model; The step of performing feature fusion on the coded data to be extended and the coded data of the reference picture based on the relative position relationship between the target picture and the reference picture through the target model to obtain target coded data includes: By using the first target model, feature transformation is performed on the relative position relationship between the target image and the reference image and the coding data of the reference image to obtain reference coding data; the reference coding data carries image features of each position in the reference image under the viewing angle of the target image; By using the second target model, performing an iterative k-times image expansion process on the to-be-expanded coded data based on the reference coded data, to obtain the target coded data; The i-th image expansion process includes: Using the reference coded data, image information generation processing is performed on the ith input coded data to obtain the ith generated overall expression; i, k are positive integers, i is less than or equal to k; Performing an i-th noise addition process on the coded data to be expanded to obtain an i-th original overall expression; fusing the i-th generated overall expression and the i-th original overall expression based on the mask image to obtain i-th output encoded data; the mask image carries the relative position relationship between the target image and the display image; Among them, the first input coded data is obtained by performing the first noise addition process on the coded data of the target image; the i-th output coded data is the i+1-th input coded data; and the target coded data is the k-th output coded data.

5. The method according to claim 3, wherein encoding the target picture of the first resolution to obtain the encoded data to be expanded comprises: Performing encoding processing on the target picture of the first resolution to obtain encoded data of the target picture; Based on the expansion requirement information corresponding to the display image, the encoding data of the target image is resized to obtain the encoding data to be expanded; the encoding data to be expanded corresponds to the size of the target image.

6. The method according to claim 2, wherein the parsing result further includes a relative position relationship between the target image and the reference image; The method further comprises: Decoding the encoded data of the reference picture to obtain a compressed reference picture; Based on the extended requirement information corresponding to the display picture and the relative position relationship between the target picture and the reference picture, aligning and cropping the compressed reference picture to obtain a compressed display picture; Enlarging the resolution of the compressed display image to a target resolution to obtain the display image; or; The method further comprises: Decoding the encoded data of the reference picture to obtain a compressed reference picture; Enlarging the resolution of the compressed reference picture to a target resolution to obtain a target reference picture; Based on the extended demand information corresponding to the display picture and the relative position relationship between the target picture and the reference picture, the target reference picture and the target picture are subjected to image alignment, cropping and image fusion processing to generate the display picture.

7. The method according to claim 6, further comprising: In response to obtaining the compressed reference picture, analyzing the compressed reference picture; When the analysis result indicates that the compressed reference picture contains target content that does not meet the expansion requirement, the target content in the compressed reference picture is modified.

8. An image processing method, the method comprising: In response to a photographing instruction, obtaining a target picture and a corresponding reference picture including the same object; The shooting position of the target picture is the same as that of the reference picture, the shooting direction of the target picture is the same as that of the reference picture, and the field of view angle of the target picture is smaller than that of the reference picture; Saving the reference image and the target image; The target image is displayed, and the reference image is used to generate a display image based on a target model together with the target image in response to an expansion instruction for the target image. The size of the display image is larger than the size of the target image, and the display image includes the image content of the target image and the expanded content.

9. The method according to claim 8, characterized in that The storing of the reference picture and the target picture comprises: Saving the reference picture to the header file of the target picture; or, Performing encoding processing on the reference picture to obtain encoding data of the reference picture; The encoded data of the reference picture is saved in the header file of the target picture.

10. An electronic device comprising: A display screen for displaying pictures; A processing device comprising: The first acquisition component is used to obtain the target image; A second obtaining component is used to obtain a reference picture, wherein the shooting position of the target picture is the same as that of the reference picture, the shooting direction of the target picture is the same as that of the reference picture, and the field of view angle of the target picture is smaller than that of the reference picture; A generating component, configured to generate a display image based on a target model, the reference image, and the target image; wherein the size of the display image is larger than the size of the target image, the display image includes image content of the target image and extended content, and the extended content is display content generated based on the target model, the reference image, and the target image for extending at least one direction of the target image; The display screen is used to display the target image.

Images