Image processing method and electronic device

Abstract

The embodiment of the present application provides an image processing method and an electronic device, which can be applied to an electronic device, wherein the method comprises: obtaining a first image and a second image of a target scene, wherein the first image is an image of the target scene taken by a first camera, and the second image is an image of the target scene taken by a second camera; performing feature extraction on the first image to obtain a first feature map of the first image, and performing feature extraction on the second image to obtain a second feature map of the second image; determining a first image feature and a second image feature according to the first feature map and the second feature map; the first image feature is an image feature participating in feature matching, and the second image feature is an image feature not participating in feature matching; generating a disparity map between the first image and the second image according to the first image feature and the second image feature. The accuracy of the disparity map can be improved by the embodiment of the present application.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method and electronic device. Background Art

[0002] Stereo vision matching technology is one of the most important research directions in the field of computer vision. The main goal of stereo vision matching is to find the corresponding pixel in one image from two images of the same scene, and then calculate the disparity of the pixel based on the pixel and its corresponding pixel. Finally, the disparity map between the two images can be generated according to the disparity of each pixel, and the disparity map between the two images can be used for depth calculation of subsequent images.

[0003] However, in practical applications, due to the influence of factors such as repeated texture or weak texture, stereo vision matching may fail. For example, taking two images as an example, a pixel in one image may find multiple corresponding pixels in the other image, resulting in low accuracy of the generated disparity map. Summary of the invention

[0004] The present application provides an image processing method and an electronic device, which can improve the accuracy of a disparity map, thereby helping to improve the accuracy and reliability of depth calculation.

[0005] In a first aspect, the present application provides an image processing method, which is applied to an electronic device, and the method includes: acquiring a first image and a second image of a target scene, the first image being an image of the target scene taken by a first camera, and the second image being an image of the target scene taken by a second camera; performing feature extraction on the first image to obtain a first feature map of the first image, and performing feature extraction on the second image to obtain a second feature map of the second image; determining a first image feature and a second image feature based on the first feature map and the second feature map; wherein the first image feature is an image feature participating in feature matching, and the second image feature is an image feature not participating in feature matching; and generating a disparity map between the first image and the second image based on the first image feature and the second image feature.

[0006] It can be seen that by determining the image features participating in feature matching and the image features not participating in feature matching in the feature maps of two images of the same scene taken by two different cameras, the accuracy of feature matching can be improved, thereby improving the accuracy of the disparity map.

[0007] Wherein, there are repeated textures and / or weak textures in the target scene. That is, both the first image and the second image include repeated textures and / or weak textures, and repeated textures and weak textures may cause feature matching errors during feature matching. The image processing method provided in the first aspect is implemented to determine the first image feature and the second image feature in the feature map corresponding to the two images. The first image feature may refer to an image feature in the first image that is different from other positions. Such image features can accurately match the corresponding image features in the second image when performing feature matching, so the first image feature is an image feature participating in feature matching; the second image feature may be an image feature corresponding to repeated textures or weak textures in the first image, that is, the second image feature is an image feature in the first image that cannot be distinguished from other positions, that is, the second image feature may be the same or similar to the image features at other positions in the first image, and such image features may match multiple image features in the second image when performing feature matching. In order to reduce the feature matching errors caused by such image features, the second image feature is an image feature that does not participate in feature matching. It can be understood that the first image feature can also refer to an image feature in the second image that is different from other positions. When performing feature matching, such image features can accurately match the corresponding image features in the first image, so the first image feature is the image feature participating in feature matching; the second image feature can be an image feature corresponding to repeated texture or weak texture in the second image, that is, the second image feature is an image feature in the second image that cannot be distinguished from other positions, that is, the second image feature may be the same or similar to the image features at other positions in the second image. When performing feature matching, such image features may match multiple image features in the first image. In order to reduce the feature matching errors caused by such image features, the first image feature is an image feature that does not participate in feature matching. By determining the first image feature and the second image feature, such image features corresponding to repeated texture or weak texture can be determined as image features that do not participate in feature matching, thereby reducing the feature matching error rate of repeated texture and weak texture scenes in the process of feature matching of image features in two images, thereby improving the accuracy of the disparity map.

[0008] In combination with the method provided in the first aspect, in some embodiments, obtaining the first image and the second image of the target scene may be receiving a touch operation on a shooting control in a camera application; in response to the touch operation, calling the first camera to shoot the target scene to obtain the first image, and calling the second camera to shoot the target scene to obtain the second image. That is, during the user's photo-taking process, the shooting control in the camera application may be triggered, and the electronic device may respond to the touch operation and call two different cameras to shoot the same scene to obtain the first image and the second image, and a disparity map between the two images may be generated in real time and quickly.

[0009] In combination with the method provided in the first aspect, in some embodiments, the first feature map includes N image features; the N image features include the i-th image feature, i, N are positive integers, and i is less than or equal to N; according to the first feature map and the second feature map, determining the first image feature and the second image feature can be: determining M candidate image features from the second feature map, the image position of each candidate image feature in the M candidate image features is on the same horizontal line as the image position of the i-th image feature; determining the feature matching participation of the i-th image feature according to the i-th image feature and the M candidate image features; determining the i-th image feature as the first image feature or the second image feature according to the feature matching participation of the i-th image feature. By determining the candidate image features that are on the same horizontal line as the image position of the i-th image feature, the computational complexity of the feature matching participation of the image features can be reduced, and then the feature matching participation of each image feature can be used to determine whether each image feature is the first image feature or the second image feature, which is beneficial to the accuracy of the subsequent disparity map.

[0010] In combination with the method provided in the first aspect, in some embodiments, the feature matching participation of the ith image feature is determined based on the ith image feature and M candidate image features, including: determining M correlations based on the ith image feature and the M candidate image features; one correlation is the correlation between the ith image feature and one of the M candidate image features; determining a first correlation and a second correlation from the M correlations; the first correlation is the maximum correlation among the M correlations, and the second correlation is the maximum correlation among the M correlations except the first correlation; determining the feature matching participation of the ith image feature based on the first correlation and the second correlation.

[0011] Wherein, determining the feature matching participation of the i-th image feature according to the first correlation and the second correlation includes: in response to the ratio between the first correlation and the second correlation being less than the participation threshold, determining the feature matching participation of the i-th image feature to be a first value; in response to the ratio between the first correlation and the second correlation being greater than or equal to the participation threshold, determining the feature matching participation of the i-th image feature to be a second value; wherein the first value is different from the second value. By comparing whether the ratio between the first correlation and the second correlation is less than the participation threshold, it can be determined whether the two candidate image features corresponding to the i-th image feature are similar, that is, whether the i-th image feature is a repeated texture or a weak texture. If the ratio between the first correlation and the second correlation is less than the participation threshold, it means that the i-th image feature is a repeated texture or a weak texture, and the feature matching participation of the i-th image feature can be determined to be the first value, and the first value can be used to indicate that the i-th image feature does not participate in feature matching; if the ratio between the first correlation and the second correlation is greater than or equal to the participation threshold, it means that the i-th image feature is not a repeated texture or a weak texture, and the feature matching participation of the i-th image feature can be determined to be the second value, and the second value can be used to indicate that the i-th image feature participates in feature matching.

[0012] In combination with the method provided in the first aspect, in some embodiments, determining the i-th image feature as the first image feature or the second image feature according to the feature matching participation of the i-th image feature includes: determining the i-th image feature as the second image feature in response to the feature matching participation of the i-th image feature being a first value; determining the i-th image feature as the first image feature in response to the feature matching participation of the i-th image feature being a second value. The feature matching participation can be used to determine whether each image feature is a weak texture or a repeated texture, thereby reducing the feature matching error rate and improving the accuracy of the disparity map.

[0013] In combination with the method provided in the first aspect, in some embodiments, the disparity processing model is used to perform feature extraction on the first image to obtain a first feature map of the first image, and to perform feature extraction on the second image to obtain a second feature map of the second image, and to determine the first image feature and the second image feature based on the first feature map and the second feature map, and to generate a disparity map between the first image and the second image based on the first image feature and the second image feature; the first image feature is an image feature that participates in feature matching, and the second image feature is an image feature that does not participate in feature matching. The disparity processing model can be used to more accurately determine the image features that do not participate in feature matching and the image features that participate in feature matching, thereby improving the accuracy of disparity map generation.

[0014] Among them, the disparity processing model includes an encoding module and a decoding module. The encoding module is used to extract features from the first image to obtain a first feature map of the first image, and to extract features from the second image to obtain a second feature map of the second image. The first image feature and the second image feature are determined according to the first feature map and the second feature map. The first image feature is an image feature participating in feature matching, and the second image feature is an image feature not participating in feature matching. The encoding module can more accurately determine the image features participating in feature matching and the image features not participating in feature matching. The decoding module is used to generate a disparity map between the first image and the second image according to the first image feature and the second image feature. The first image feature and the second image feature are processed by the decoding module, so that the decoding module can perform feature matching on the first image feature. At the same time, without performing feature matching on the second image feature, the decoding module can focus on the global image features of the first image and the second image, which can greatly improve the accuracy of the disparity map.

[0015] In conjunction with the method provided in the first aspect, in some embodiments, a deployment parameter of a disparity processing model is received; and the disparity processing model is deployed according to the deployment parameter of the disparity processing model. It should be understood that the disparity processing model can be a small model that occupies less memory resources, and by using such a model, the disparity processing model can be quickly deployed directly on any electronic device according to the deployment parameter of the disparity processing model.

[0016] In combination with the method provided in the first aspect, in some embodiments, a first sample image and a second sample image are obtained; an initial disparity processing model is called to perform feature extraction on the first sample image to obtain a first sample feature map of the first sample image, and feature extraction is performed on the second sample image to obtain a second sample feature map of the second sample image; first sample image features and second sample image features are determined based on the first sample feature map and the second sample feature map; the first sample image features are image features that participate in feature matching, and the second sample image features are image features that do not participate in feature matching; a sample disparity map between the first sample image and the second sample image is generated based on the first sample image features and the second sample image features; an initial disparity processing model is trained based on the sample disparity map between the first sample image and the second sample image to obtain a disparity processing model. By training the model, a disparity processing model that generates a more accurate disparity map can be obtained.

[0017] In a second aspect, the present application provides an electronic device, comprising one or more processors and one or more memories; wherein the one or more memories are coupled to the one or more processors, and the one or more memories are used to store computer program code, and the computer program code includes computer instructions, and when the one or more processors execute the computer instructions, the electronic device executes the method described in the first aspect and any possible implementation method of the first aspect.

[0018] In a third aspect, the present application provides a computer-readable storage medium comprising instructions, which, when executed on an electronic device, causes the electronic device to execute the method described in the first aspect and any possible implementation of the first aspect.

[0019] In a fourth aspect, the present application provides a computer program product comprising instructions, which, when executed on an electronic device, enables the electronic device to execute the method described in the first aspect and any possible implementation of the first aspect.

[0020] It can be understood that the electronic device provided in the second aspect, the computer storage medium provided in the third aspect, and the computer program product provided in the fourth aspect are all used to execute the method provided in the first aspect of the present application. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1a is a schematic diagram of weak texture provided in an embodiment of the present application;

[0022] Figure 1b is a schematic diagram of a repeated texture provided in an embodiment of the present application;

[0023] Figure 1c-1d This is a schematic diagram of the disparity map effect obtained by adopting the first method provided in the embodiment of the present application;

[0024] Figure 1e-1f is a schematic diagram of a disparity map effect obtained by using an image processing method provided in an embodiment of the present application;

[0025] Figure 2a-2d is a set of user interface schematic diagrams provided in an embodiment of the present application;

[0026] Figure 3 It is a schematic diagram of the software architecture provided by the embodiment of the present application;

[0027] Figure 4 It is a flowchart of an image processing method provided in an embodiment of the present application;

[0028] Figure 5is a schematic diagram of candidate image features of the i-th image feature provided by the implementation of this application;

[0029] Figure 6a-6c is a schematic diagram of the training process of the disparity processing model provided in an embodiment of the present application;

[0030] Figure 7 It is a hardware structure diagram of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0031] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application.

[0032] 1. Texture

[0033] Texture is usually used to describe the surface features of an object. For example, texture can be a pattern, wallpaper, etc. In an image, texture is a structure formed by the repeated arrangement of a certain pattern. Texture is a very important image feature. It not only reflects the grayscale statistical information of the image, but also reflects the spatial distribution information and structural information of the image. It has a wide range of application scenarios in computer vision and image recognition. Generally speaking, an image can include rich textures, which may include weak textures and repeated textures.

[0034] The so-called weak texture refers to the lack of obvious texture on the surface of the object in the image or the complex and unclear texture pattern; for example, Figure 1a In the image shown, the weak texture may be a wall 11 in the image. When extracting image features, generally no effective image features can be extracted from the weak texture in the image, or effective image features exist but similar or identical image features exist.

[0035] The so-called repeated texture refers to the texture that appears repeatedly in a certain pattern in the image. Generally, effective image features can be extracted from the repeated texture area, but there may be multiple identical or similar image features. For example, Figure 1b In the image shown, the repeated texture may be area 12 in the image. During feature matching, for repeated textures, when two cameras are called by an electronic device to simultaneously shoot the same scene, two images are obtained, and the image features of one image may have multiple identical or similar image features in the other image.

[0036] For weak textures and repeated textures, when a binocular camera (i.e., two cameras installed in an electronic device) captures two images and performs feature matching on the image features in the two images, feature matching errors are likely to occur.

[0037] 2. Disparity and Disparity Map

[0038] Disparity refers to the positional deviation of the same image feature in two images. The positional deviation of multiple image features in an image in two images can constitute a disparity map between the two images, that is, the disparity map between the two images is determined by two images taken by cameras at two different positions in an electronic device. Specifically, the disparity map is based on any one of the two images and is composed of the disparity between each image feature in the image and its corresponding image feature in the other image. In other words, the disparity map can be a two-dimensional image used to store the disparity values ​​of all image features of a single view.

[0039] 3. Loss Parameters

[0040] The loss parameter refers to the difference between the predicted value (estimated value) and the actual value (expected value, reference value, label (ground truth)). "Loss" means the penalty imposed on the model for failing to produce the expected results. Its function is to determine the performance of the model by comparing the predicted output and the expected output of the model, and then find the optimization direction. If the deviation between the two is very large, the loss value will be large; if the deviation is small or the values ​​are almost the same, the loss value will be very low. Among them, the loss parameter can be the loss value of the loss function (Loss Function) or the loss value of the cost function (Cost Function). For a model, if its loss parameter meets the corresponding conditions, then the model can be deployed and / or used.

[0041] At present, in a stereoscopic vision scene, method 1 can be used to perform stereoscopic vision matching on two images of the same scene to obtain a disparity map between the two images. The specific process of method 1 includes: for two images of the same scene (such as a first image and a second image), image features that match each image feature in the first image can be determined in the second image, and then the disparity of each image feature in the first image is calculated based on each image feature in the first image and the image feature that matches it, and finally the disparity map between the first image and the second image can be generated based on the time difference of each image feature. For example, for Figure 1a As for the weak texture in the image shown in FIG. 1 , the disparity map between the first image and the second image can be finally generated as follows: Figure 1c shown. Figure 1c The area 13 in the figure represents the parallax effect corresponding to the weak texture. Figure 1b As for the repeated texture in the image shown in FIG. 1 , the disparity map between the first image and the second image can be finally generated as follows: Figure 1d shown. Figure 1d The area 14 in the figure represents the parallax effect corresponding to the repeated texture.

[0042] The image processing method provided in the embodiment of the present application can also perform stereoscopic visual matching on two images of the same scene to obtain a disparity map between the two images. Specifically, the image processing method provided in the embodiment of the present application can perform feature extraction on two images taken of the same scene to obtain feature maps corresponding to the two images respectively, and determine the image features that participate in feature matching and the image features that do not participate in feature matching based on the feature maps corresponding to the two images, and generate a disparity map between the two images based on the image features that participate in feature matching and the image features that do not participate in feature matching. Exemplarily, for Figure 1a As for the weak texture in the image shown in FIG. 1 , the first image and the second image are processed by the image processing method provided in the embodiment of the present application, and the disparity map between the first image and the second image is finally generated as shown in FIG. Figure 1e As shown, Figure 1e The area 15 in the figure represents the parallax effect corresponding to the weak texture. Figure 1b As for the repeated texture in the image shown in FIG. 1 , the first image and the second image are processed by the image processing method provided in the embodiment of the present application, and the disparity map between the first image and the second image is finally generated as shown in FIG. Figure 1f As shown, Figure 1f Area 16 in the figure represents the parallax effect corresponding to the repeated texture.

[0043] For weak textures in images, Figure 1e The parallax effect corresponding to the weak texture represented by area 15 in Figure 1c The parallax effect corresponding to the weak texture represented by area 13 in FIG. 1 is for the repeated texture in the image. Figure 1f The parallax effect corresponding to the repeated texture represented by area 16 in Figure 1d The parallax effect corresponding to the repeated texture represented by the area 14 in FIG. It can be seen that the parallax map finally generated by the image processing method provided in the embodiment of the present application is more accurate than the parallax map generated by the method 1.

[0044] Next, the application scenarios of the embodiments of the present application are introduced. The image processing method and electronic device provided in the embodiments of the present application can be applied to scenes where depth calculation of the captured images is required using at least two cameras (such as binocular cameras) or other hardware, and can also be applied to scenes where the captured images are blurred with a large aperture. The image processing method provided in the embodiments of the present application can extract features of two images of the same scene captured by two different cameras in the electronic device when there are weak textures or repeated textures in the image, and obtain feature maps corresponding to the two images respectively, and determine the image features participating in feature matching and the image features not participating in feature matching according to the feature maps corresponding to the two images, thereby reducing the situation of feature matching errors, and then according to the image features participating in feature matching and the image features not participating in feature matching, a disparity map between the two images can be generated, and the image features participating in feature matching and the image features not participating in feature matching can reduce the problem of inaccurate feature matching caused by weak textures and repeated textures through the image features participating in feature matching and the image features not participating in feature matching, and the accuracy of the disparity map can be improved to a certain extent.

[0045] It should be understood that the image processing method provided in the embodiment of the present application can be applied to electronic devices equipped with at least two cameras and having the image processing capability. The above-mentioned electronic devices (i.e., electronic device 200) may include but are not limited to: mobile phones, tablet computers, desktop computers, laptop computers, etc.

[0046] By implementing the image processing method provided in the embodiment of the present application, the electronic device 200 can obtain the first image and the second image of the target scene; illustratively, the electronic device includes a first camera and a second camera. In some embodiments, the electronic device 200 can call the first camera to shoot the target scene to obtain the first image, and call the second camera to shoot the target scene to obtain the second image. According to the first feature map of the first image and the second feature map of the second image, the image features participating in feature matching and the image features not participating in feature matching are determined. By determining the image features participating in feature matching and the image features not participating in feature matching, it can help reduce the influence of factors such as repeated textures or weak textures on feature matching. According to the first image features and the second image features, a disparity map between the first image and the second image is generated. By improving the accuracy of feature matching, the accuracy of the disparity map between the first image and the second image can be improved. The subsequent embodiments are described by taking an electronic device configured with a binocular camera (i.e., configured with two cameras, such as a first camera and a second camera) as an example.

[0047] The electronic device is not limited to a mobile phone or a tablet computer. The electronic device 200 may also be a desktop computer, a laptop computer, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a cellular phone, a personal digital assistant (PDA), an augmented reality (AR) device, a virtual reality (VR) device, an artificial intelligence (AI) device, a wearable device, an in-vehicle device, a smart home device, and / or a smart city device. The embodiment of the present application does not impose any special restrictions on the specific type of the electronic device.

[0048] Figure 2a-2d An exemplary set of user interfaces displayed on the electronic device 200 is shown below. Figure 2a-2d The application scenarios of the image processing method provided by the embodiments of the present application are specifically introduced.

[0049] first, Figure 2a The user interface of the electronic device 200 showing the installed applications, i.e., the home page, is shown as an example. Figure 2a As shown, one or more application icons are displayed in the main page, such as a "clock" application icon, a "calendar" application icon, a "weather" application icon, and the like.

[0050] The one or more application icons include an icon of a "camera" application, namely, icon 211. The electronic device 200 can detect a user operation on icon 211, such as a click operation. In response to the user operation, the electronic device 200 can turn on the camera, call the camera to capture an image, and display the image. Figure 2b The user interface shown.

[0051] In one implementation, the lock screen user interface of the electronic device 200 displays a "camera" icon. The electronic device 200 can detect a user operation on the "camera" icon, such as a click and pull-up operation to the top of the lock screen user interface. In response to the user operation, the electronic device 200 can turn on the camera, call the camera to capture an image, and display the image. Figure 2b The user interface shown.

[0052] When the electronic device 200 turns on the camera, it can turn on at least two cameras and call the at least two cameras to shoot images of the same scene, that is, the at least two cameras are used to shoot images of the same scene from different perspectives. For ease of understanding, the embodiment of the present application is described by turning on two cameras of the electronic device. Assuming that the electronic device includes a first camera and a second camera, the first camera in the electronic device 200 shoots an image of a portrait scene from a first perspective, and the second camera in the electronic device 200 shoots an image of the same portrait scene from a second perspective, and there is a difference between the first perspective and the second perspective.

[0053] Figure 2b and Figure 2c The user interface when the “camera” application is running on the electronic device 200 is exemplarily shown. The user interface may include a window 221, a shooting control 222, an image display control 223, and a reversal control 224.

[0054] Among them, the window 221 is used to display the image captured by the camera, which can be understood as the display area of ​​the image captured by the camera, so that the user can input a click operation to the shooting control 222 or the reversal control 224 according to the picture displayed in the window 221. In some embodiments, the window 221 also includes some controls, such as a focus adjustment control. When the electronic device 200 detects a user operation acting on the focus adjustment control, it can adjust the focus of the camera in response to the user operation, for example, from 1x to 2x, and the type of the camera is not limited here.

[0055] When one camera is turned on by default in the electronic device 200, the image displayed in the window 221 is the image taken by the camera. When two cameras are turned on in the electronic device 200, the image displayed in the window 221 may be an image obtained by fusion or reconstruction of the disparity map between two images of the same scene, and the image is an image with depth and color information; that is, the disparity map between two images taken from different perspectives by two cameras can be used to fuse or reconstruct the first image and the second image to obtain the image of the scene, and finally the image of the scene can be displayed in the window 221. When the electronic device 200 turns on two cameras, the effect of the image displayed in the window 221 is relatively more realistic. Optionally, a blurred image can be displayed in the window 221, and the blurred image refers to an image obtained by performing depth calculation based on the disparity map to obtain a depth map, and blurring the image based on the depth map.

[0056] When the electronic device 200 detects a user operation acting on the shooting control 222, in response to the user operation, the shooting image can be determined from the collected images, and a thumbnail of the shooting image can be displayed in the image display control 223. In one implementation, the electronic device 200 includes a first camera and a second camera, and a thumbnail of a first image shot by the first camera or a thumbnail of a second image shot by the second camera can be displayed in the image display control 223, or an image with depth and color information can be displayed in the image display control 223, and the image is obtained by fusing or reconstructing the first image and the second image according to a disparity map between the first image and the second image; or a blurred image can be displayed in the image display control 223.

[0057] Then, when a user operation acting on the image display control 223 is detected, in response to the user operation, Figure 2d The user interface shown in the figure displays an image with depth and color information or a blurred image; in addition, the image with depth and color information and the blurred image can also be saved in a gallery, and then when a click operation acting on the image with depth and color information or the blurred image in the gallery is detected, in response to the click operation, Figure 2d The user interface shown displays an image with depth and color information or a blurred image.

[0058] When the electronic device 200 detects a user operation on the inversion control 224, in response to the user operation, it can turn off the rear camera and turn on the front camera, and call the front camera to capture an image; or, it can turn off the front camera and turn on the rear camera, and call the rear camera to capture an image.

[0059] like Figure 2b and Figure 2c The user interface shown also includes a "night scene" mode, a "portrait" mode, a "photo" mode, and a "video recording" mode. These modes are used for example and do not constitute a limitation on the embodiments of the present application. For example, in actual applications, a "multi-lens video recording" mode is also included. Among them, the "video recording" mode is used to record video files, and the "multi-lens video recording" mode is used to record video files when the front camera and the rear camera are turned on at the same time. The "night scene" mode is used to capture night scene images and improve the clarity of night scene images. The "portrait" mode is mainly used to capture portrait images. The above-mentioned "night scene" mode, "portrait" mode and other modes can all adopt the image processing method provided in the embodiments of the present application. The "photo" mode is used to capture a single frame image. Figure 2b and Figure 2c In the video, the mode selected by the user is the "Portrait" mode.

[0060] When the electronic device 200 detects a user operation acting on the image display control 223, in response to the user operation, the electronic device 200 displays the following Figure 2d The user interface shown in FIG. 2 is a user interface for browsing pictures, and may include an image display window 231 for displaying images or videos.

[0061] The user interface may also include a control 234 , a share control 235 , a favorite control 236 , an edit control 233 , a delete control 237 , and the like.

[0062] Control 234 may be used to display detailed information of the image, such as shooting time, shooting location, color encoding format, bit rate, frame rate, pixel size, etc.

[0063] The sharing control 235 can be used to send the displayed image or video to other applications for use. For example, after detecting a user operation on the sharing control, in response to the user operation, the electronic device 200 can display one or more application icons, and the one or more application icons include the icon of social software 1. After detecting a user operation on the application icon of social software 1, in response to the user operation, the electronic device 200 can send the image or video to the social software 1, and further, the user can share the above image or video with friends through the social software.

[0064] The favorites control 236 can be used to mark an image or video. Figure 2d In the user interface shown, when a user operation acting on the favorites control is detected, in response to the user operation, the electronic device 200 may mark the displayed image or video as a user's favorite image or video. The electronic device 200 may generate an album for displaying the images or videos marked as the user's favorite.

[0065] The delete control 237 may be used to delete the displayed image or video.

[0066] The electronic device 200 is described in detail below. Figure 2d The specific process of the user interface is shown.

[0067] first, Figure 3 The software architecture of the electronic device 200 is exemplarily shown.

[0068] The software system of the electronic device 200 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. In the embodiment of the present invention, the Android system of the layered architecture is taken as an example to exemplify the software structure of the electronic device 200.

[0069] The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into five layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system library, the hardware abstraction layer (HAL), and the kernel layer.

[0070] The application layer can include a series of application packages. Figure 3 As shown, the application package may include camera, gallery, video, music, navigation, calendar, map, WLAN and other applications. In the embodiment of the present application, the camera can provide the following during operation: Figure 2b to Figure 2d After starting the camera, the electronic device 200 can call the camera to capture an image. In one implementation, the electronic device 200 includes at least a first camera and a second camera, for example, the first camera is a main camera, and the second camera is a telephoto camera, and the first camera and the second camera can capture the same scene from different perspectives.

[0071] The application framework layer provides an application programming interface (API) and a programming framework for the application of the application layer. The application framework layer includes some predefined functions. In an embodiment of the present application, the application framework layer includes a camera service framework, and the camera service framework may include a disparity processing model. Optionally, the disparity processing model performs feature extraction on the first image of the target scene to obtain a first feature map of the first image, and performs feature extraction on the second image of the target scene to obtain a second feature map of the second image, and then determines the first image feature and the second image feature according to the first feature map and the second feature map, the first image feature is the image feature participating in feature matching, and the second image feature is the image feature not participating in feature matching; based on the first image feature and the second image feature, a disparity map between the first image and the second image is generated. The disparity processing model can realize the functions of image feature extraction, determination of image features participating in feature matching and not participating in feature matching, and disparity map generation.

[0072] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0073] Android Runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system. The core library contains two parts: one is the function that the Java language needs to call, and the other is the Android core library.

[0074] The system library can include multiple functional modules. For example: surface manager, media library, 3D graphics processing library (for example: OpenGL), etc. The surface manager is used to manage the display subsystem and provide fusion of 2D and 3D layers for multiple applications. The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc. The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, synthesis, and layer processing, etc.

[0075] The hardware abstraction layer (HAL) is an interface layer between the kernel layer and the hardware, which can be used to control the actions of the hardware. The hardware abstraction layer may include a hardware module for controlling the actions of the hardware. Optionally, the hardware abstraction layer may include a disparity processing model.

[0076] The kernel layer is the foundation of the Android system. For example, ART relies on the kernel layer to perform underlying functions, such as threads and low-level memory management. The kernel layer is a layer between hardware and software. The kernel layer includes at least a display driver, a camera driver, an audio driver, a sensor driver, a GPU driver, etc. In an embodiment of the present application, the kernel layer includes a camera driver and a display driver. The camera driver is used to drive the camera to capture images, and the display driver is used to drive the display screen to have images with depth and color information or blurred images.

[0077] based on Figure 3 The software architecture shown in the figure illustrates the image processing method provided by the embodiment of the present application. In some embodiments, the electronic device 200 detects a touch operation acting on a shooting control in a camera application, and in response to the touch operation, starts the camera so that the camera shoots an image. Upon detecting the touch operation, the camera application triggers an instruction to start the camera. The camera application calls the API interface of the application framework layer to send the instruction to the camera service framework, and the camera service framework calls the hardware abstraction layer to send the instruction to the camera driver. The camera driver can drive the camera to start and drive the camera to shoot images. The images shot by the camera can be cached in the image buffer area.

[0078] When the electronic device 200 starts the camera, at least two cameras can be started. The embodiment of the present application takes starting the first camera and the second camera in the electronic device as an example. Exemplarily, the first camera is the main camera and the second camera can be a telephoto camera. When the electronic device 200 detects the touch operation, it immediately starts the first camera and the second camera. Optionally, the disparity processing model is deployed in the camera service framework, and the first camera can transmit the captured first image to the disparity processing model in the camera service framework, and the second camera can transmit the captured second image to the disparity processing model in the camera service framework. The disparity processing model processes the first image and the second image to obtain a disparity map between the first image and the second image. The camera service framework can perform image depth calculation on the disparity map, and construct an image of the target scene based on the calculated depth map. The camera service framework calls the hardware abstraction layer to transmit the constructed image of the target scene to the display screen, so that the display screen displays the image of the target scene. Optionally, the disparity processing model is deployed in the hardware abstraction layer. The first camera can transmit the captured first image to the disparity processing model in the hardware abstraction layer, and the second camera can transmit the captured second image to the disparity processing model in the hardware abstraction layer. The disparity processing model processes the first image and the second image to obtain a disparity map between the first image and the second image.

[0079] Figure 4 The flowchart of an image processing method is exemplarily shown, and the image processing method can be executed by the electronic device 200. Figure 2a-2d The user interface shown and Figure 3 The software architecture of the electronic device 200 is shown. Below, the embodiment of the present application will specifically introduce the process of image processing by the electronic device 200.

[0080] 401, the electronic device 200 obtains a first image and a second image of a target scene, where the first image is an image of the target scene taken by a first camera, and the second image is an image of the target scene taken by a second camera.

[0081] Among them, the target scene can be any scene, such as a portrait scene, a landscape scene, and the like. The shooting parameters of the first camera and the second camera are different, such as different focal lengths, etc. The first camera and the second camera are respectively deployed at different positions of the electronic device 200. The target scene in the first image and the second image generally has rich texture information, and the texture information may include weak textures, repeated textures, and the like. Exemplarily, the embodiment of the present application takes the first image and the second image as images of shooting the target portrait as an example, the target portrait is any portrait, the first image is obtained by shooting the target portrait with the first camera, and the second image can be obtained by shooting the target portrait with the second camera.

[0082] Optionally, the first image and the second image may be pre-photographed and stored in a gallery, and the electronic device 200 may obtain the first image and the second image of the target scene from the gallery. Optionally, the first image and the second image may be obtained by other devices calling the first camera and the second camera to photograph the target scene and stored in the gallery, or the first image and the second image may be obtained by other devices calling the first camera and the second camera to photograph the target scene and send to the electronic device 200.

[0083] In another implementation, the first image and the second image may be captured in real time, and the electronic device 200 receives a touch operation on a capture control in a camera application, wherein the capture control is a control in the camera application, and the electronic device 200 opens the camera application and detects whether a touch operation on the capture control is received; when the touch operation on the capture control is detected, in response to the touch operation, the first camera is called to capture the target scene to obtain the first image, and the second camera is called to capture the target scene to obtain the second image. The touch operation may be a click operation, a slide operation, etc. For example, the electronic device 200 may receive a touch operation on the capture control. Figure 2b A click operation of the shooting control 222 in the user interface is shown. Upon receiving the click operation of the shooting control, the electronic device 200 may call the first camera to shoot the target scene to obtain a first image, and call the second camera to shoot the target scene to obtain a second image.

[0084] Optionally, the first camera and the second camera simultaneously shoot the target scene from different perspectives. The electronic device 200 can call the first camera to shoot the target scene from a first perspective to obtain a first image, and at the same time, the electronic device 200 can call the second camera to shoot the target scene from a second perspective to obtain a second image, and the first perspective and the second perspective are different. By shooting the target scene from different perspectives, the depth information or position information of the object in the target scene can be obtained.

[0085] Optionally, when the electronic device 200 receives a user operation acting on a camera application, the camera application is started. Starting the camera application can trigger the start of the first camera and the second camera, thereby calling the first camera and the second camera to shoot the target scene. That is, before the touch operation acting on the shooting control is detected, the first camera and the second camera can always shoot the target scene, and cache the image shot by the first camera and the image shot by the second camera in the image cache area. The touch operation acting on the shooting control is detected, and in response to the touch operation, a first image is determined from at least one image shot by the first camera cached in the image cache area, and a second image is determined from at least one image shot by the second camera cached in the image cache area. The first image and the second image refer to images obtained by shooting the target scene at the same time. Starting the camera application can be done by Figure 2a The user operation trigger shown may also be triggered by a user operation on other applications, for example, by a user operation on a shooting icon of an instant messaging application.

[0086] When the electronic device 200 detects a touch operation on the shooting control, the camera application triggers a shooting instruction. The camera application calls the API interface of the application framework layer to send the shooting instruction to the camera service framework, and the camera service framework calls the camera interface to read the cached image from the image buffer area, and determines the first image and the second image from the cached image.

[0087] 402, the electronic device 200 performs feature extraction on the first image to obtain a first feature map of the first image, and performs feature extraction on the second image to obtain a second feature map of the second image.

[0088] In one implementation, the disparity processing model includes an encoding module. Calling the encoding module in the disparity processing model can extract features from the first image to obtain a first feature map of the first image, and calling the encoding module to extract features from the second image to obtain a second feature map of the second image.

[0089] Optionally, before executing step 402, the first image and the second image can be corrected to obtain the corrected first image and the second image. The correction process can make the first image and the second image located in the same plane and parallel to each other, that is, each row of image features of the first image and the second image are colinear (that is, on a horizontal line), which is conducive to the subsequent search for candidate image features for image features and the calculation of correlation.

[0090] 403, the electronic device 200 determines a first image feature and a second image feature according to the first feature map and the second feature map; the first image feature is an image feature that participates in feature matching, and the second image feature is an image feature that does not participate in feature matching.

[0091] For example, in Figure 4 In the first feature map and the second feature map, both include image features. Figure 4 A rectangle shown represents an image feature. When calculating the disparity map, it is necessary to first find matching image features in the two images, that is, for an image feature in one image, it is necessary to find a matching image feature in the other image. In this process, the image feature may find multiple matching image features in the other image, which will cause errors in the subsequently generated disparity map. Therefore, in an embodiment of the present application, the image features participating in feature matching can be determined based on the first feature map and the second feature map. This type of image feature refers to image features that are rich and different from those in other positions. At the same time, image features that do not participate in feature matching are also determined. This type of image feature refers to image features corresponding to repeated textures or weak textures, that is, image features that may find multiple matching image features in the feature matching process. Exemplarily, in Figure 4 In , the image feature marked as 0 is the determined second image feature, and the image feature marked as 1 is the determined first image feature.

[0092] Among them, the first image feature and the second image feature may come from the first feature map, that is, determining the first image feature and the second image feature according to the first feature map and the second feature map may be: determining the first image feature and the second image feature in the first feature map according to the second feature map and the first feature map. The first feature map includes N image features, and the N image features include the i-th image feature, i, N are positive integers, and i is less than or equal to N; M candidate image features are determined from the second feature map, M is an integer greater than 1, and the image position of each candidate image in the M candidate image features is on the same horizontal line as the image position of the i-th image feature. Based on the i-th image feature and the M candidate image features, the feature matching participation of the i-th image feature is determined, and the feature matching participation can be used to indicate whether the i-th image feature participates in feature matching. According to the feature matching participation of the i-th image feature, the i-th image feature is determined to be the first image feature or the second image feature.

[0093] The image position of each of the M candidate image features and the image position of the ith image feature are on the same horizontal line, which means that: in the coordinate systems constructed with the same position of the first feature map and the second feature map as the origin, the ordinate of the image position of each candidate image is the same as the ordinate of the image position of the ith image feature. For example, a coordinate system is constructed with the lower left vertex of the first feature map as the origin, and similarly, a coordinate system is constructed with the lower left vertex of the second feature map as the origin; for another example, a coordinate system is constructed with the upper right vertex of the second feature map as the origin, and similarly, a coordinate system is constructed with the upper right vertex of the second feature map as the origin. Exemplarily, as Figure 5 FIG. 1 is a schematic diagram of determining candidate image features provided by an embodiment of the present application. Figure 5 Each small rectangle shown represents an image feature. Let the i-th image feature be Figure 5 In the image feature 61, three candidate image features can be found on the same horizontal line, namely Figure 5 Image features 62-image features 64 in.

[0094] Optionally, the image position of each candidate image in the M candidate image features and the image position of the i-th image feature may be on the same vertical line. That is, in a coordinate system constructed with the same position of the first feature map and the second feature map as the origin, the horizontal coordinate of the image position of each candidate image is the same as the horizontal coordinate of the image position of the i-th image feature.

[0095] In one implementation, the feature matching participation degree may be determined based on the correlation between the i-th image feature and the M candidate image features. Determining the feature matching participation degree of the i-th image feature based on the i-th image feature and the M candidate image features may include the following steps:

[0096] Step 1: Determine M correlations based on the i-th image feature and the M candidate image features; one correlation is the correlation between the i-th image feature and one of the M candidate image features. That is, the correlation between the i-th image feature and each candidate image feature can be calculated respectively. For example, the above Figure 5 In the example, the correlation between the image feature 61 and the image feature 62, the correlation between the image feature 61 and the image feature 63, and the correlation between the image feature 61 and the image feature 64 can be calculated respectively, and finally three correlations can be obtained.

[0097] Step 2: determine a first correlation and a second correlation from the M correlations; the first correlation is the maximum correlation among the M correlations, and the second correlation is the maximum correlation among the M correlations except the first correlation.

[0098] In one implementation, the M correlations may be sorted in descending order to obtain sorted M correlations, and the first two correlations are selected from the sorted M correlations, where the first two correlations are the first correlation and the second correlation.

[0099] Step 3: Determine the feature matching participation of the i-th image feature according to the first correlation and the second correlation.

[0100] It should be understood that the reason for selecting the maximum correlation and the second largest correlation among the M correlations here is that if the conditions between the two correlations are met, then it can be considered that the i-th image feature may find at least two image features that match it in the second feature map, that is, there may be repeated textures or weak textures at the image position where the i-th image feature is located or in the second image at this point. At this time, the feature matching participation of the i-th image feature can be determined as a first value, and the first value is used to indicate that the i-th image feature does not participate in feature matching. In this way, the feature matching error can be reduced to improve the accuracy of the subsequently generated disparity map.

[0101] In one implementation, it is determined whether the difference between the first correlation and the second correlation is less than a threshold value. If the difference between the first correlation and the second correlation is less than the threshold value (that is, the first correlation and the second correlation meet the conditions), the feature matching participation of the i-th image feature is determined to be a first value. If the difference between the first correlation and the second correlation is greater than or equal to the threshold value, the feature matching participation of the i-th image feature is determined to be a second value. The threshold value can be determined according to requirements. For example, the threshold value can be set to 0.1, 0.5, etc.

[0102] In another implementation, in response to the ratio between the first correlation and the second correlation being less than the participation threshold, the feature matching participation of the i-th image feature is determined to be a first value; in response to the ratio between the first correlation and the second correlation being greater than or equal to the participation threshold, the feature matching participation of the i-th image feature is determined to be a second value. That is, the ratio between the first correlation and the second correlation is calculated, and it is determined whether the ratio is less than the participation threshold. If the ratio is less than the participation threshold, that is, the first correlation / the second correlation<the participation threshold, then the feature matching participation of the i-th image feature is determined to be the first value. If the ratio is greater than or equal to the participation threshold, that is, the first correlation / the second correlation≥the participation threshold, then the feature matching participation of the i-th image feature is determined to be the second value. The participation threshold can be set according to demand, such as the participation threshold can be set to 2, 3, etc., and the embodiments of the present application do not limit this.

[0103] The first value is different from the second value. The first value is used to indicate that the i-th image feature does not participate in feature matching, and the second value is used to indicate that the i-th image feature participates in feature matching. The first value and the second value can be set as required. For example, the first value can be 0 and the second value can be 1.

[0104] In one implementation, determining the i-th image feature as the first image feature or the second image feature according to the feature matching participation of the i-th image feature includes: determining the i-th image feature as the second image feature in response to the feature matching participation of the i-th image feature being a first value; and determining the i-th image feature as the first image feature in response to the feature matching participation of the i-th image feature being a second value. It should be understood that any image feature in the first feature map can be determined as the first image feature or the second image feature with reference to the i-th image feature.

[0105] Optionally, the first image feature and the second image feature may come from the second feature map, that is, determining the first image feature and the second image feature according to the first feature map and the second feature map may be: determining the first image feature and the second image feature in the second feature map according to the second feature map and the first feature map. The second feature map includes N1 image features, and the N1 image features include the jth image feature, j, N1 are positive integers, and j is less than or equal to N1; M1 candidate image features are determined from the first feature map, M1 is an integer greater than 1, and the image position of each candidate image in the M1 candidate image features is on the same horizontal line as the image position of the jth image feature. Based on the jth image feature and the M1 candidate image features, the feature matching participation of the jth image feature is determined, and the feature matching participation can be used to indicate whether the jth image feature participates in feature matching, and based on the feature matching participation of the jth image feature, the jth image feature is determined to be the first image feature or the second image feature. How to determine the jth image feature as the first image feature or the second image feature can refer to the specific implementation method of determining the ith image feature as the first image feature or the second image feature mentioned above, which will not be repeated here.

[0106] That the image position of each candidate image in the M1 candidate image features is on the same horizontal line as the image position of the j-th image feature means that in a coordinate system constructed with the same position of the first feature map and the second feature map as the origin, the ordinate of the image position of each candidate image is the same as the ordinate of the image position of the j-th image feature. Optionally, the image position of each candidate image in the M1 candidate image features and the image position of the j-th image feature may also be on the same vertical line.

[0107] 404 , the electronic device 200 generates a disparity map between the first image and the second image according to the first image feature and the second image feature.

[0108] In one implementation, the electronic device 200 may perform feature matching on the image features in the first feature map and the second feature map that need to participate in feature matching according to the first image feature and the second image feature, and determine the disparity corresponding to the first image feature based on the feature matching result, and generate a disparity map between the first image and the second image according to the disparity corresponding to the first image feature. The disparity map between the first image and the second image may be as follows: Figure 4 shown.

[0109] In another implementation, the electronic device 200 includes a disparity processing model, and the disparity processing model is used to generate a disparity map between the first image and the second image according to the first image feature and the second image feature. Optionally, the disparity processing model can be used to perform steps 402-404, that is, the first image and the second image are input into the disparity processing model to obtain a disparity map between the first image and the second image output by the disparity processing model. Optionally, the disparity processing model can be used to perform step 402, and to perform step 404. That is, the disparity processing model is used to perform feature extraction on the first image to obtain a first feature map of the first image, and to perform feature extraction on the second image to obtain a second feature map of the second image. The disparity processing model is also used to generate a disparity map between the first image and the second image according to the first image feature and the second image feature.

[0110] Optionally, a disparity processing model is deployed in the camera service framework, and the camera service framework can call the disparity processing model to process the first image and the second image to obtain a disparity map between the first image and the second image. The processing here includes: extracting features from the first image to obtain a first feature map of the first image, and extracting features from the second image to obtain a second feature map of the second image, determining first image features and second image features based on the first feature map and the second feature map; and generating a disparity map between the first image and the second image based on the first image features and the second image features. Among them, the disparity processing model can be a small model that occupies less memory resources and has fewer model parameters, and the small model can be a CNN (Convolutional Neural Networks) model, etc. Optionally, the HAL layer deploys a disparity processing model, and the HAL layer can call the disparity processing model to process the first image and the second image to obtain a disparity map between the first image and the second image.

[0111] The disparity processing model can be a model deployed and run in different types of electronic devices. The disparity processing model may include an encoding module and a decoding module. The encoding module is used to extract features from the first image to obtain a first feature map of the first image, and to extract features from the second image to obtain a second feature map of the second image. According to the first feature map and the second feature map, the first image feature and the second image feature are determined. Optionally, the encoding module may be used only to extract the first image and the second image. The decoding module is used to generate a disparity map between the first image and the second image according to the first image feature and the second image feature, that is, the first image feature and the second value of the first image feature, the second image feature and the first value of the second image feature can be input into the decoding module to obtain the disparity map between the first image and the second image output by the decoding module. By inputting the second value of the first image feature and the first value of the second image feature into the decoding module together, it is possible to guide the decoding module which image features do not participate in feature matching and which image features participate in feature matching in the process of generating the disparity map, so that the finally generated disparity map is more accurate. Optionally, the first image feature and the second image feature are from a first feature map, and the first image feature and the second value of the first image feature, the second image feature and the first value of the second image feature, and the second feature map can be input into a decoding module. Optionally, the first image feature and the second image feature are from a second feature map, and the first image feature and the second value of the first image feature, the second image feature and the first value of the second image feature, and the first feature map can be input into a decoding module.

[0112] In an embodiment of the present application, a first image and a second image of a target scene are obtained, wherein the first image is an image of the target scene taken by a first camera, and the second image is an image of the target scene taken by a second camera; feature extraction is performed on the first image to obtain a first feature map of the first image, and feature extraction is performed on the second image to obtain a second feature map of the second image; first image features and second image features are determined based on the first feature map and the second feature map; the first image features are image features that participate in feature matching, and the second image features are image features that do not participate in feature matching; a disparity map between the first image and the second image is generated based on the first image features and the second image features. It can be seen that by determining the image features that participate in feature matching and the image features that do not participate in feature matching in the feature maps of two images of the same scene taken by two different cameras, the accuracy of feature matching can be improved, thereby improving the accuracy of the disparity map.

[0113] Among them, the disparity processing model deployed by the electronic device 200 may come from the server. That is to say, in the process of training the model, the server can obtain a disparity processing model that meets the deployment conditions, and send the deployment parameters of the disparity processing model to the electronic device 200, so that the electronic device 200 receives these deployment parameters and deploys the disparity processing model based on these deployment parameters, so that the electronic device 200 can call the disparity processing model to process the first image and the second image, and obtain the disparity map between the first image and the second image. Optionally, the electronic device 200 can train the model by itself, and deploy the disparity processing model after meeting the deployment conditions. The so-called deployment condition refers to that the model loss is lower than a certain value (such as a preset value).

[0114] The training process of the disparity processing model is described below.

[0115] Training method 1: directly train the initial disparity processing model. Specifically, obtain a first sample image and a second sample image; call the initial disparity processing model to extract features from the first sample image to obtain a first sample feature map of the first sample image, and extract features from the second sample image to obtain a second sample feature map of the second sample image; determine the first sample image feature and the second sample image feature according to the first sample feature map and the second sample feature map; the first sample image feature is an image feature that participates in feature matching, and the second sample image feature is an image feature that does not participate in feature matching; generate a sample disparity map between the first sample image and the second sample image according to the first sample image feature and the second sample image feature; train the initial disparity processing model according to the sample disparity map between the first sample image and the second sample image to obtain a disparity processing model.

[0116] Training method 2: When the disparity processing model is a small model that occupies less memory resources and has fewer model parameters, in order to improve the accuracy of the disparity map, the embodiment of the present application can be trained through three stages to obtain the disparity processing model. Figure 6a-6c Flowchart shown. Figure 6a-6c The flowchart shown may include:

[0117] Phase 1: See Figure 6a , the third feature map of each third sample image obtained by feature extraction using the large model is used to perform distillation training on the original encoding module in the original disparity processing model.

[0118] s1, pre-training the large model: obtain a sample image set, and use the sample image set to perform self-supervisory training on the large model to obtain a third sample feature map of each third sample image in the sample image set. Optionally, the sample image set can be used to perform self-supervisory training on the encoding module in the large model. Among them, the sample image set can be imageNet, etc. The large model here refers to a model that occupies more memory resources and has more model parameters. Exemplarily, the large model can be a transformer model, a moco (momentum contrast) model, etc.

[0119] s2, on the above sample image set, the third sample feature map of each third sample image obtained by feature extraction using the large model is used to perform distillation training on the original coding module in the original disparity processing model to obtain an initial coding module. Specifically, the original coding module in the original disparity processing model is called to perform feature extraction on each third sample image in the sample image set to obtain a fourth sample feature map of each third sample image in the sample image set; then, based on the third sample feature map and the fourth sample feature map of each third sample image, the loss value between the third sample feature map and the fourth sample feature map in each third sample image is determined, and based on the loss value, the parameters of the original coding module are adjusted to obtain an initial coding module.

[0120] Phase 2, see Figure 6b , fix the trained large model parameters, and use the binocular data (such as the fourth sample image and the fifth sample image) to train the original decoding module in the original disparity processing model.

[0121] s3. Obtain a fourth sample image and a fifth sample image. The fourth sample image and the fifth sample image are training images obtained by photographing the same scene from different perspectives using two cameras (such as binocular cameras). For example, both are training images of a target portrait. The fourth sample image may be one that is photographed by one of the two cameras, and the fifth sample image may be one that is photographed by the other of the two cameras. It is understandable that the fourth sample image and the fifth sample image have the same image identifier (identifier, ID), and the image ID can be used to identify images of the same scene photographed from different perspectives using two cameras. Images with the same image ID mean two images taken using a binocular camera, and the two images are images obtained by photographing the same scene. However, the shooting parameters are different, such as shooting angle, focal length, color, etc.

[0122] s4, fix the feature extraction parameters in the large model, call the large model to perform feature extraction on the fourth sample image and the fifth sample image, respectively, to obtain a fourth sample feature map of the fourth sample image and a fifth sample feature map of the fifth sample image. Optionally, call the encoding module in the large model to perform feature extraction on the fourth sample image and the fifth sample image, respectively, to obtain a fourth sample feature map of the fourth sample image and a fifth sample feature map of the fifth sample image.

[0123] s5, according to the fifth sample feature map and the sixth sample feature map, determine the third sample image feature and the fourth sample image feature, wherein the third sample image feature is a sample image feature that participates in feature matching, and the fourth sample image feature is a sample image feature that does not participate in feature matching. According to the fifth sample feature map and the sixth sample feature map, the third sample image feature and the fourth sample image feature can be determined by referring to the determination method of the first image feature and the second image feature, which will not be repeated here.

[0124] s6. Use the third sample image feature and the fourth sample image feature to train the original decoding module in the original disparity processing model to obtain an initial decoding module. That is, the third sample image feature and the fourth sample image feature are input into the original decoding module in the original disparity processing model to obtain a disparity map between the fourth sample image and the fifth sample image output by the original decoding module. According to the disparity map between the fourth sample image and the fifth sample image, the original decoding module is trained to obtain an initial decoding module. Optionally, when determining the third sample image feature and the fourth sample image feature, the second value (such as 1) of the third sample image feature and the first value (such as 0) of the fourth sample image feature can be input into the original decoding module to train the original decoding module.

[0125] Phase 3: See Figure 6c , the initial disparity processing model consisting of the initial encoding module and the initial decoding module is trained end-to-end.

[0126] s7. Obtain a first sample image and a second sample image. The first sample image and the second sample image are training images obtained by photographing the same scene from different perspectives using two cameras (such as binocular cameras). For example, both are training images of a target portrait. The first sample image may be a training image of the target portrait photographed by one of the binocular cameras, and the second sample image may be a training image of the target portrait photographed by another of the binocular cameras. The first sample image and the second sample image may be the same as the fourth sample image and the fifth sample image, or may be different.

[0127] s8. Calling the initial encoding module in the initial disparity processing model to perform feature extraction on the first sample image to obtain a first sample feature map of the first sample image, and to perform feature extraction on the second sample image to obtain a second sample feature map of the second sample image.

[0128] s9. Determine the first sample image feature and the second sample image feature according to the first sample feature map and the second sample feature map; the first sample image feature is the image feature participating in feature matching, and the second sample image feature is the image feature not participating in feature matching.

[0129] s10, calling the initial decoding module in the initial disparity processing model, and generating a sample disparity map between the first sample image and the second sample image according to the first sample image feature and the second sample image feature. Specifically, the first sample image feature and the second sample image feature can be input into the initial decoding module to obtain a sample disparity map between the first sample image and the second sample image output by the initial decoding module.

[0130] s11. According to the sample disparity map between the first sample image and the second sample image, the initial disparity processing model is trained to obtain the disparity processing model. Specifically, the first sample image and the second sample image correspond to disparity map labels, and the model loss (i.e., loss parameter) of the initial disparity processing model is determined according to the sample disparity map and the disparity map label, and the parameters of the initial encoding module and the parameters of the initial decoding module in the initial disparity processing model are adjusted in the direction of reducing the model loss to obtain the disparity processing model.

[0131] In summary, in the process of model training, since the large model has more parameters, its performance is better than that of the small model. Therefore, by introducing the large model as a reference for training the disparity processing model, the final disparity processing model can reach the performance of the large model, and can better complete the feature extraction of the image and improve the accuracy of the disparity map; at the same time, the disparity processing model can be deployed in any electronic device, especially some electronic devices with less memory resources.

[0132] It should be understood that s1 to s11 may be performed by the server. When the above-mentioned model loss satisfies the deployment condition, the server may send the deployment parameters of the disparity processing model to the electronic device 200, so that the electronic device 200 deploys the disparity processing model based on the deployment parameters. Optionally, after deploying the disparity processing model, the electronic device 200 may input the first image and the second image into the disparity processing model, and the disparity processing model may output a disparity map between the first image and the second image.

[0133] In the above, the server performs model training and sends the deployment parameters of the disparity processing model that meets the deployment conditions to the electronic device as an example. In addition to the server, the PC can also perform model training and send the deployment parameters of the disparity processing model that meets the deployment conditions to the electronic device. Other devices can also perform model training, which is not limited in the embodiments of the present application.

[0134] Optionally, the deployment parameters of the disparity processing model may also be preset on the electronic device. For example, the deployment parameters of the disparity processing model may be preset when the electronic device leaves the factory, so that when the electronic device is turned on or the camera application is used for the first time, the disparity processing model may be deployed based on the deployment parameters of the disparity processing model; or, when the electronic device receives a trigger operation to deploy the disparity processing model, the disparity processing model may be deployed based on the deployment parameters of the disparity processing model; and so on.

[0135] After deploying the disparity processing model, the electronic device may receive update parameters or patches from the server, where the update parameters are used to update the disparity processing model, and the patches are used to repair or upgrade the disparity processing model. Optionally, after deploying the disparity processing model, the electronic device may repair the disparity processing model by itself. For example, when the system of the electronic device is updated, the electronic device may repair the disparity processing model by itself.

[0136] Figure 7 The hardware structure diagram of the electronic device 200 is exemplarily shown.

[0137] The electronic device 200 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio processing module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, at least two cameras 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0138] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the electronic device 200. In other embodiments of the present application, the electronic device 200 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0139] The processor 110 may include one or more processing units, for example, the processor 110 may include an AP, a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated into one or more processors. In an embodiment of the present application, the processor 110 may deploy a disparity processing model and call the disparity processing model.

[0140] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0141] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0142] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0143] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present invention is only a schematic illustration and does not constitute a structural limitation on the electronic device 200. In other embodiments of the present application, the electronic device 200 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0144] The charging management module 140 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through a wireless charging coil of the electronic device 200. While the charging management module 140 is charging the battery 142, it may also power the electronic device through the power management module 141.

[0145] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the display screen 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0146] The wireless communication function of the electronic device 200 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0147] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the electronic device 200 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of the antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0148] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 200. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0149] The modulation and demodulation processor may include a modulator and a demodulator. The modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is transmitted to the application processor.

[0150] The application processor outputs sound signals through an audio device (not limited to the speaker 170A, the receiver 170B, etc.), or displays images or videos through the display screen 194. In some embodiments, the modem processor can be an independent device. In other embodiments, the modem processor can be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0151] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 200. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering processing, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0152] In some embodiments, the antenna 1 of the electronic device 200 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 200 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology, etc. GNSS may include the global positioning system (GPS), the global navigation satellite system (GLONASS), the Beidou navigation satellite system (BDS), the quasi-zenith satellite system (QZSS) and / or the satellite based augmentation system (SBAS).

[0153] The electronic device 200 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information. In an embodiment of the present application, the display screen 194 is used to display a feature map, or the display screen can also display an image with depth and color information and a blurred image.

[0154] The internal memory 121 may include one or more random access memories (RAM) and one or more non-volatile memories (NVM).

[0155] Random access memory may include static random-access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM, for example, the fifth generation DDR SDRAM is generally referred to as DDR5 SDRAM), etc. Non-volatile memory may include disk storage devices and flash memory.

[0156] The random access memory can be directly read and written by the processor 110, and can be used to store executable programs (such as machine instructions) of the operating system or other running programs, and can also be used to store user and application data.

[0157] The non-volatile memory may also store executable programs and user and application data, etc., and may be loaded into the random access memory in advance for direct reading and writing by the processor 110 .

[0158] The external memory interface 120 can be used to connect to an external non-volatile memory to expand the storage capacity of the electronic device 200. The external non-volatile memory communicates with the processor 110 through the external memory interface 120 to implement a data storage function. For example, files such as music and videos are stored in the external non-volatile memory.

[0159] The electronic device 200 can implement audio functions such as music playing and recording through the audio processing module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone interface 170D, and the application processor.

[0160] The audio processing module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio processing module 170 can also be used to encode and decode audio signals. In some embodiments, the audio processing module 170 can be arranged in the processor 110, or some functional modules of the audio processing module 170 can be arranged in the processor 110.

[0161] Speaker 170A is also called "speaker". Electronic device 200 can listen to music or listen to hands-free calls through speaker 170A. Receiver 170B is also called "earpiece". When electronic device 200 answers a call or voice message, the voice can be answered by placing receiver 170B close to the human ear. Microphone 170C is also called "microphone" or "microphone". When making a call or sending a voice message, the user can speak by placing the human mouth close to microphone 170C to input the sound signal into microphone 170C.

[0162] The earphone interface 170D is used to connect a wired earphone and can be a USB interface 130 or a 3.5 mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0163] The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be set on the display screen 194. The gyroscope sensor 180B can be used to determine the motion posture of the electronic device 200. The air pressure sensor 180C is used to measure the air pressure. The magnetic sensor 180D includes a Hall sensor. The electronic device 200 can use the magnetic sensor 180D to detect the opening and closing of the flip leather case. The acceleration sensor 180E can detect the magnitude of the acceleration of the electronic device 200 in various directions (generally three axes). The distance sensor 180F is used to measure the distance. The proximity light sensor 180G can include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The ambient light sensor 180L is used to sense the brightness of the ambient light. The fingerprint sensor 180H is used to collect fingerprints. The temperature sensor 180J is used to detect the temperature. The touch sensor 180K, also known as a "touch device". The touch sensor 180K can be disposed on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen". The touch sensor 180K is used to detect a touch operation on or near it. The bone conduction sensor 180M can obtain a vibration signal.

[0164] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The electronic device 200 may receive key input and generate key signal input related to user settings and function control of the electronic device 200.

[0165] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. Indicator 192 can be an indicator light, which can be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

[0166] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to or disconnected from the electronic device 200 by inserting the SIM card interface 195 or pulling the SIM card interface 195 out. The electronic device 200 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, etc.

[0167] The term "user interface (UI)" in the specification, claims and drawings of this application refers to the medium interface for interaction and information exchange between an application or operating system and a user, which realizes the conversion between the internal form of information and the form acceptable to the user. The user interface of an application is source code written in a specific computer language such as Java and extensible markup language (XML). The interface source code is parsed and rendered on the terminal device, and finally presented as content that the user can recognize, such as pictures, text, buttons and other controls. Controls, also known as widgets, are the basic elements of the user interface. Typical controls include toolbars, menu bars, text boxes, buttons, scroll bars, pictures and text. The properties and contents of controls in the interface are defined by tags or nodes, such as XML through <textview> 、 <imgview> 、 <videoview>The controls contained in the interface are specified by nodes such as . A node corresponds to a control or attribute in the interface. After parsing and rendering, the node is presented as user-visible content. In addition, many applications, such as hybrid applications, usually also contain web pages in their interfaces. A web page, also known as a page, can be understood as a special control embedded in the application interface. A web page is source code written in a specific computer language, such as hypertext markup language (GTML), cascading style sheets (CSS), JavaScript (JS), etc. The web page source code can be loaded and displayed as user-recognizable content by a browser or a web page display component with similar functions to a browser. The specific content contained in a web page is also defined by tags or nodes in the web page source code. For example, GTML is defined by 、 、 <video> 、 <canvas>To define the elements and attributes of a web page.

[0168] The most common form of user interface is graphical user interface (GUI), which refers to a user interface related to computer operation that is displayed in a graphical manner. It can be an icon, window, control or other interface element displayed on the display screen of an electronic device, where a control can include icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets and other visual interface elements.

[0169] As used in the specification and appended claims of the present application, the singular expressions "a", "an", "above", "the" and "the" are intended to include plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in the present application refers to and includes any or all possible combinations of one or more of the listed items. As used in the above embodiments, the term "when..." may be interpreted to mean "if..." or "after..." or "in response to determining..." or "in response to detecting...", depending on the context. Similarly, the phrases "when determining..." or "if (the stated condition or event) is detected" may be interpreted to mean "if it is determined..." or "in response to determining..." or "when (the stated condition or event) is detected" or "in response to detecting (the stated condition or event)", depending on the context.

[0170] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loading and executing computer program instructions on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server, a data center, etc. that contains one or more available media integration. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk), etc.

[0171] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.< / canvas> < / video> < / videoview> < / imgview> < / textview>

Claims

1. An image processing method, characterized in that: The method is applied to an electronic device, and the method comprises: Acquire a first image and a second image of a target scene, wherein the first image is an image of the target scene taken by a first camera, and the second image is an image of the target scene taken by a second camera; and the target scene contains repeated textures and / or weak textures; Performing feature extraction on the first image to obtain a first feature map of the first image, and performing feature extraction on the second image to obtain a second feature map of the second image; Determine a first image feature and a second image feature according to the first feature map and the second feature map; the first image feature is an image feature participating in feature matching, and the first image feature refers to an image feature in the first image or the second image that is distinguished from other positions; the second image feature is an image feature not participating in feature matching, and the second image feature refers to an image feature corresponding to a repeated texture or a weak texture in the first image or the second image; A disparity map between the first image and the second image is generated according to the first image feature and the second image feature.

2. The method according to claim 1, characterized in that The first feature map includes N image features; the N image features include the i-th image feature, i and N are positive integers, and i is less than or equal to N; The determining the first image feature and the second image feature according to the first feature map and the second feature map includes: Determine M candidate image features from the second feature map, wherein the image position of each candidate image feature in the M candidate image features is on the same horizontal line as the image position of the i-th image feature; M is an integer greater than 1; Determining a feature matching participation degree of the i-th image feature according to the i-th image feature and the M candidate image features; According to the feature matching participation degree of the i-th image feature, it is determined that the i-th image feature is the first image feature or the second image feature.

3. The method according to claim 2, characterized in that The determining, according to the i-th image feature and the M candidate image features, the feature matching participation degree of the i-th image feature comprises: Determine M correlations according to the i-th image feature and the M candidate image features; one correlation is a correlation between the i-th image feature and one of the M candidate image features; Determine a first correlation and a second correlation from the M correlations; the first correlation is the maximum correlation among the M correlations, and the second correlation is the maximum correlation among the M correlations except the first correlation; The feature matching participation degree of the i-th image feature is determined according to the first correlation degree and the second correlation degree.

4. The method according to claim 3, characterized in that The determining, according to the first correlation and the second correlation, the feature matching participation degree of the i-th image feature includes: In response to a ratio between the first relevance and the second relevance being less than a participation threshold, determining a feature matching participation of the i-th image feature to be a first value; In response to a ratio between the first relevance and the second relevance being greater than or equal to a participation threshold, determining a feature matching participation of the i-th image feature to be a second value; Wherein, the first value is different from the second value.

5. The method according to claim 4, characterized in that The determining, according to the feature matching participation degree of the i-th image feature, that the i-th image feature is the first image feature or the second image feature includes: In response to the feature matching participation degree of the i-th image feature being the first value, determining the i-th image feature as the second image feature; In response to the feature matching participation degree of the i-th image feature being the second value, determining the i-th image feature as the first image feature.

6. The method according to claim 1, characterized in that The step of acquiring the first image and the second image of the target scene comprises: Receive touch actions for capture controls in the camera app; In response to the touch operation, the first camera is called to shoot the target scene to obtain the first image, and the second camera is called to shoot the target scene to obtain the second image.

7. The method according to any one of claims 1 to 6, characterized in that: The electronic device includes a disparity processing model, which is used to perform feature extraction on the first image to obtain a first feature map of the first image, and to perform feature extraction on the second image to obtain a second feature map of the second image, and to determine first image features and second image features based on the first feature map and the second feature map, and to generate a disparity map between the first image and the second image based on the first image features and the second image features; the first image features are image features that participate in feature matching, and the second image features are image features that do not participate in feature matching.

8. The method according to claim 7, characterized in that The disparity processing model includes an encoding module and a decoding module. The encoding module is used to perform feature extraction on the first image to obtain a first feature map of the first image, and to perform feature extraction on the second image to obtain a second feature map of the second image; determine a first image feature and a second image feature according to the first feature map and the second feature map; the first image feature is an image feature that participates in feature matching, and the second image feature is an image feature that does not participate in feature matching; The decoding module is used to generate a disparity map between the first image and the second image according to the first image feature and the second image feature.

9. The method according to claim 7, characterized in that The method further comprises: receiving deployment parameters of a disparity processing model; The disparity processing model is deployed according to the deployment parameters of the disparity processing model.

10. The method according to claim 7, characterized in that The method further comprises: Acquire a first sample image and a second sample image; Calling an initial disparity processing model to perform feature extraction on the first sample image to obtain a first sample feature map of the first sample image, and performing feature extraction on the second sample image to obtain a second sample feature map of the second sample image; Determine a first sample image feature and a second sample image feature according to the first sample feature map and the second sample feature map; the first sample image feature is an image feature that participates in feature matching, and the second sample image feature is an image feature that does not participate in feature matching; generating a sample disparity map between the first sample image and the second sample image according to the first sample image feature and the second sample image feature; The initial disparity processing model is trained according to a sample disparity map between the first sample image and the second sample image to obtain the disparity processing model.

11. An electronic device, characterized in that: The method comprises one or more processors and one or more memories; wherein the one or more memories are coupled to the one or more processors, and the one or more memories are used to store computer program codes, wherein the computer program codes include computer instructions, and when the one or more processors execute the computer instructions, the method as described in any one of claims 1 to 10 is executed.

12. A chip system, characterized in that: The chip system is applied to an electronic device, and the chip system includes one or more processors, and the processor is used to call computer instructions so that the electronic device executes the method as described in any one of claims 1-10.

13. A computer-readable storage medium comprising instructions, characterized in that: When the instructions are executed on an electronic device, the method according to any one of claims 1 to 10 is executed.

Images