Image processing methods, apparatus, storage media and electronic devices

Abstract

This specification discloses an image processing method, apparatus, storage medium, and electronic device. The method includes: acquiring a first grayscale image collected by a structured light module and a calibration reference image corresponding to the structured light module; determining an offset mapping relationship for the calibration reference image based on the first grayscale image; and performing depth correction processing on the structured light module based on the offset mapping relationship and the calibration reference image in the depth calculation scenario of the structured light module.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to an image processing method, apparatus, storage medium and electronic device. Background Technology

[0002] With the rapid development of computer technology, structured light technology is being applied more and more widely. Typically, structured light technology first projects structured light, such as multiple fixed sinusoidal templates, onto the object being measured in the scene. The structured light deforms on the object's surface, and then an image is captured. Finally, by calculating the deformation of the structured light in the scene or the correspondence between the structured light and points on the image, depth calculation algorithms such as triangulation are used to accurately calculate the scene's depth information. Summary of the Invention

[0003] This specification provides an image processing method, apparatus, storage medium, and electronic device, the technical solutions of which are as follows:

[0004] Firstly, this specification provides an image processing method, the method comprising:

[0005] Acquire a first grayscale image captured by the structured light module, and acquire a calibration reference image corresponding to the structured light module, wherein the calibration reference image is a grayscale image generated by the structured light module at a calibration distance from the structured light module;

[0006] Based on the first grayscale image, determine the offset mapping relationship for the calibration reference image;

[0007] In the depth calculation scenario, the target offset for the calibration reference image is determined based on the offset mapping relationship, and the structured light module is subjected to depth correction processing based on the target offset.

[0008] Secondly, this specification provides an image processing apparatus, the apparatus comprising:

[0009] The image acquisition module is used to acquire a first grayscale image collected by the structured light module, and to acquire a calibration reference image corresponding to the structured light module. The calibration reference image is a grayscale image generated by the structured light module at a calibration distance from the structured light module.

[0010] The relationship determination module is used to determine the offset mapping relationship for the calibration reference image based on the first grayscale image;

[0011] The depth calculation module is used to determine the target offset for the calibration reference image based on the offset mapping relationship in the depth calculation scene, and to perform depth correction processing on the structured light module based on the target offset.

[0012] Thirdly, this specification provides a computer storage medium storing a plurality of instructions adapted for loading by a processor and executing the above-described method steps.

[0013] Fourthly, this specification provides an electronic device that may include: a processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and to execute the above-described method steps.

[0014] The beneficial effects of the technical solutions provided in some embodiments of this specification include at least the following:

[0015] In one or more embodiments of this specification, by acquiring a first grayscale image and a calibration reference image based on the structured light module, and determining the offset mapping relationship with respect to the calibration reference image based on the first grayscale image, in the depth calculation scenario, the target offset amount relative to the calibration reference image can be determined based on the offset mapping relationship, and then the structured light module can be subjected to depth correction processing. This can significantly reduce or even eliminate the impact of uncertainties such as vibration, deformation, and temperature on the depth calculation scenario, overcome the large offset of the calibration reference image in the actual scenario, improve the offset robustness of depth calculation, and ensure the quality of depth calculation. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in this specification or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a scene diagram of an image processing system provided in this manual;

[0018] Figure 2 This is an example diagram illustrating an image processing method provided in this specification;

[0019] Figure 3 This is a schematic diagram of a deep solution application scenario provided in this manual;

[0020] Figure 4 This is a flowchart illustrating another image processing method provided in this manual;

[0021] Figure 5 This is a schematic diagram of the structure of an image processing device provided in this specification;

[0022] Figure 6This is a structural diagram of a relationship determination module provided in this specification;

[0023] Figure 7 This is a schematic diagram of the structure of a correlation detection unit provided in this specification;

[0024] Figure 8 This is a schematic diagram of the structure of an electronic device provided in this specification;

[0025] Figure 9 This is a schematic diagram of the operating system and user space provided in this manual;

[0026] Figure 10 yes Figure 9 Architecture diagram of the Android operating system in China;

[0027] Figure 11 yes Figure 9 Architecture diagram of the iOS operating system. Detailed Implementation

[0028] The technical solutions in this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0029] In the description of this application, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. In the description of this application, it should be noted that, unless otherwise expressly specified and limited, "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances. Furthermore, in the description of this application, unless otherwise stated, "multiple" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist; for example, A and / or B can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship.

[0030] The present application will now be described in detail with reference to specific embodiments.

[0031] Please see Figure 1This is a scene diagram of an image processing system provided in this specification. Figure 1 As shown, the image processing system may include at least a client cluster and a service platform 100.

[0032] The client cluster may include at least one client, such as Figure 1 As shown, it specifically includes client 1 corresponding to user 1, client 2 corresponding to user 2, ..., client n corresponding to user n, where n is an integer greater than 0.

[0033] Each client in a client cluster can be an electronic device with communication capabilities, including but not limited to: time and attendance machines, access control devices, wearable devices, handheld devices, personal computers, tablets, in-vehicle devices, smartphones, computing devices, or other processing devices connected to a wireless modem. Electronic devices may have different names in different networks, such as: user equipment, access terminal, user unit, user station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication equipment, user agent or user device, cellular phone, cordless phone, personal digital assistant (PDA), and electronic devices in 5G networks or future evolved networks.

[0034] The service platform 100 can be a standalone server device, such as a rack-mount, blade, tower, or cabinet-type server device, or a workstation, mainframe, or other hardware device with strong computing power; or it can be a server cluster composed of multiple servers. The servers in the service cluster can be composed in a symmetrical manner, wherein each server is functionally and hierarchically equivalent in the transaction chain, and each server can provide services independently. The independent provision of services can be understood as not requiring the assistance of other servers.

[0035] In one or more embodiments of this specification, the service platform 100 can establish a communication connection with at least one client in the client cluster, and complete the data interaction in the depth calculation scenario during image processing based on the communication connection, such as the data interaction of depth images in the depth calculation scenario. For example, the client can execute an image processing method to determine the target offset and can request the service platform 100 to assist in the depth correction of the structured light module based on the target offset. It is understood that for depth calculation scenarios with large computational processing volume, the client can request the service platform 100 to assist in executing the image processing method.

[0036] It should be noted that the service platform 100 establishes a communication connection with at least one client in the client cluster via a network for interactive communication. This network can be a wireless network or a wired network. Wireless networks include, but are not limited to, cellular networks, wireless LANs, infrared networks, or Bluetooth networks. Wired networks include, but are not limited to, Ethernet, universal serial bus (USB), or controller area networks. In one or more embodiments of the specification, technologies and / or formats including Hyper Text Markup Language (HTML), Extensible Markup Language (XML), etc., are used to represent data exchanged over the network (such as target compressed packets). Furthermore, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), and Internet Protocol Security (IPsec) can be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies can be used to replace or supplement the aforementioned data communication technologies.

[0037] The image processing system embodiments provided in this specification and the image processing methods described in one or more embodiments belong to the same concept. The execution entity corresponding to the image processing method involved in one or more embodiments of this specification can be the aforementioned service platform 100; the execution entity corresponding to the image processing method involved in one or more embodiments of this specification can also be the electronic device corresponding to the client, specifically determined based on the actual application environment. The implementation process of the image processing system embodiments can be detailed in the following method embodiments, and will not be repeated here.

[0038] based on Figure 1 The following is a detailed description of the model processing methods provided by one or more embodiments of this specification, as illustrated in the scene diagram.

[0039] Please see Figure 2 This document provides a flowchart illustrating an image processing method according to one or more embodiments of the present specification. This method can be implemented using a computer program and can run on an image processing device based on the von Neumann architecture. The computer program can be integrated into an application or run as a standalone utility application. The image processing device can be an electronic device.

[0040] Specifically, the image processing method includes:

[0041] S102: Obtain a first grayscale image for the structured light module, and obtain a calibration reference image corresponding to the structured light module, wherein the calibration reference image is a grayscale image generated by the structured light module at a calibration distance from the structured light module;

[0042] The structured light module is typically used in conjunction with electronic devices; it can be part of the electronic device or an expandable external component. The structured light module can acquire depth information from images of the surrounding environment.

[0043] The structured light module can project structured light onto the surface of an object in a known spatial direction. The module then collects the changes in the light signal caused by the object, which can be applied to depth calculation scenarios to calculate information such as the object's position and depth. The projected structured light can be understood as the collection of projected light rays in a known spatial direction.

[0044] In one or more embodiments of this specification, the structured light projected by the structured light module includes, but is not limited to, point structured light, line structured light, multi-line structured light, surface structured light, and phase-based structured light.

[0045] Schematic illustration: The structured light module can be any module structure in related technologies. For example, a common structured light module may include at least a transmitting device and a receiving device. The transmitting device emits specially modulated invisible infrared light onto the object being photographed, such as a laser projection device; the receiving device receives the invisible infrared light reflected back from the object. Furthermore, the structured light module of the electronic device can control the transmitting device to project, for example, an infrared speckle field onto the object's surface, and then control the receiving device to acquire the (speckled) grayscale image-IR image obtained after reflection from the object's surface. The electronic device can then use the (speckled) grayscale image and a calibration reference image, combined with relevant calculation parameters such as the internal / external parameters of the structured light module, to calculate the depth information of each pixel on the (speckled) grayscale image-IR image using devices such as chips or application-specific integrated circuits, thereby outputting a depth map.

[0046] In one or more embodiments of this specification, the structured light module may further include a camera for acquiring color images (typically planar images), thereby combining the color images with the aforementioned depth information to obtain 3D information of the photographed object in some embodiments.

[0047] The calibration reference image is typically generated in the calibration scenario of the structured light module. The calibration scenario of the structured light module can usually be the calibration of module data during the production or delivery stage of the structured light module. That is, before the structured light module is used with electronic devices, calibration data such as calibration reference images and internal / external parameters are set for the structured light module. In the depth calculation scenario of the electronic device, the set depth calculation algorithm is used to calculate the depth information of each pixel on the (speckle) grayscale image-IR image based on the (speckle) grayscale image collected by the structured light module for the object and the calibration reference image, combined with the internal / external parameters of the structured light module and other relevant calculation parameters: the depth information of each pixel on the (speckle) grayscale image-IR image is calculated.

[0048] As an illustration, due to objective environmental factors, electronic devices are subject to uncertainties such as vibration, deformation, and temperature. Changes in these uncertainties can significantly deviate from the depth calculation scene, leading to inaccurate calibration reference images. In related technologies, setting corresponding compensation algorithms can only mitigate the impact of changes in the internal / external parameters of the structured light module. However, the aforementioned uncertainties can cause irreversible damage to the calibration reference image. In practical applications, the corresponding compensation algorithms are unlikely to improve the irreversible damage to the calibration reference image. This could potentially severely affect the quality of depth information in subsequent depth calculation scenes due to factors such as the generation of numerous outliers.

[0049] As an illustration, since the calibration reference image is difficult for users of electronic devices to change after calibration, users of electronic devices usually do not have the ability to change or modify the calibration reference image. That is, in the actual application scenario of electronic devices, ordinary users cannot change or modify these calibration reference images.

[0050] In illustrative terms, the calibration reference image is generated during the calibration phase of the structured light module. Typically, calibration is performed at a calibration distance from the structured light module, where a grayscale image is generated by the receiving device at that distance. This grayscale image serves as the calibration reference image. In some implementations, multiple calibration reference images may be used, and all or part of these images may correspond to different calibration distances. Calibration reference images generated at different calibration distances can be applied to different depth calculation scenarios. These calibration reference images can be stored in the storage unit of an electronic device for easy retrieval during practical applications.

[0051] The first grayscale image may be a grayscale IR image acquired by the structured light module during the use of the electronic device, and the electronic device may cache these previously used first grayscale images. In some embodiments, the first grayscale image may be considered as a historical grayscale image stored by the electronic device;

[0052] In one or more embodiments of this specification, the electronic device may also record the ambient temperature when the structured light module acquires the first grayscale image, establish a mapping between the first grayscale image and the ambient temperature, and save the first grayscale image and the corresponding ambient temperature.

[0053] Understandably, electronic devices can acquire at least one stored first grayscale image as well as a pre-stored calibration reference image.

[0054] S104: Based on the first grayscale image, determine the offset mapping relationship for the calibration reference image;

[0055] The offset mapping relationship is used to correct or adjust the calibration reference image during depth calculation, and to query or determine the real-time (or actual) offset. The offset mapping relationship is set based on the actual application scenario and can be represented in the form of an offset mapping array, offset mapping list, offset mapping graph, etc. Through this offset mapping relationship for the calibration reference image, the current target offset for the calibration reference image can be obtained in the actual depth calculation scenario.

[0056] Understandably, by saving several first grayscale images and combining them with pre-stored calibration reference images, it is possible to mine the offset mapping relationship (such as offset mapping array or offset mapping list) of the calibration reference image in actual complex application scenarios. Thus, when the structured light module is used again, the target offset of the calibration reference image can be determined or queried based on the offset mapping relationship. In subsequent depth calculation scenarios, the depth calculation process of the structured light module can be corrected based on the target offset, thereby improving the accuracy of the output depth information and overcoming the interference of complex environments such as temperature, mechanical wear, and drops on depth calculation.

[0057] In a schematic manner, the electronic device can perform successive offsetting of the calibration reference image to obtain offset calibration images and record the offset amount in each round. During each round of offsetting, the image similarity parameter between the offset calibration image and the first grayscale image is calculated. The maximum similarity parameter in each round is taken, and the offset calibration image corresponding to the maximum similarity parameter is determined. The image offset between this offset calibration image and the first grayscale image is obtained. Since the calibration reference image is usually difficult for the user of the electronic device to change, this image offset can be used as a reference for offset adjustment of the calibration reference image. By analogy, when there are multiple first grayscale images, multiple image offsets relative to the calibration reference image determined based on the first grayscale images can be obtained.

[0058] Indicatively, following the above method, an offset mapping relationship for a calibration reference image is constructed based on the image offset determined by at least one first grayscale image.

[0059] In one feasible implementation, the mapping between the aforementioned image offset and its calibration reference image is taken as an offset mapping relationship for the calibration reference image, denoted as offset mapping relationship A. This offset mapping relationship A is usually a mapping between the calibration reference image and the image offset, which can be understood as the calibration reference image usually corresponding to an image offset in offset mapping relationship A.

[0060] Furthermore, in determining the offset mapping relationship A: there may be multiple image offsets determined based on multiple first grayscale images for the calibration reference image. These multiple image offsets can be aggregated to obtain the final aggregated image offset, which is then incorporated into the reference for establishing the offset mapping relationship A. The aggregation process could involve taking the average offset of the multiple image offsets as the final offset, taking the mode of the multiple image offsets as the final offset, taking the median of the multiple image offsets as the final offset, and so on.

[0061] In one feasible implementation, considering that different environmental conditions will have different effects on depth calculation based on the calibration reference image, the different environmental conditions can be further quantified, and the offset mapping relationship for the calibration reference image can be further constructed based on the first grayscale image generated by the structured light module under different environmental conditions.

[0062] Understandably, when the structured light module generates several first grayscale images, it records the corresponding environmental conditions and associates the first grayscale images with the corresponding environmental conditions at that time.

[0063] Indicatively, environmental conditions are quantified by at least one environmental condition parameter. Different environmental condition parameters such as different times, different scenes, different temperatures, and different humidity are combined and mapped to different environmental condition types. The first grayscale image generated under different environmental condition types will have different image offsets in the aforementioned manner.

[0064] Schematic, the environmental condition type corresponding to the generation of the first grayscale image can be included as a reference for the image offset of the calibration reference image. First grayscale images with different environmental condition types can determine different image offsets relative to the calibration reference image. Based on this, several offset mapping relationships between different environmental condition types and their corresponding image offsets relative to the calibration reference image can be established. This offset mapping relationship is essentially a mapping between different reference environmental condition types and their corresponding reference image offsets. Let's denote this as offset mapping relationship B. Offset mapping relationship B is typically a mapping of the image offsets relative to the calibration reference image, and it consists of different environmental condition types and their corresponding image offsets relative to the calibration reference image.

[0065] The environmental condition type is characterized in the offset mapping relationship by one or more of the relevant environmental condition parameters, such as time parameters, scene parameters, temperature parameters, and humidity parameters. Different combinations of relevant environmental condition parameters constitute different environmental condition types.

[0066] Optionally, multiple first grayscale images under similar environmental conditions (environmental parameters such as time parameters and scene parameters being the same or similar) can be used. Based on the multiple image offsets determined from these first grayscale images relative to the calibration reference image, these multiple image offsets can be aggregated to obtain the final aggregated image offset, which is then incorporated into the reference for establishing the offset mapping relationship. The aggregation process can involve taking the average offset from multiple image offsets, taking the mode of multiple image offsets as the average offset, and so on.

[0067] S106: In the depth calculation scenario, the target offset for the calibration reference image is determined based on the offset mapping relationship, and the structured light module is subjected to depth correction processing based on the target offset.

[0068] The depth calculation scenario can be understood as a scenario involving the calculation of depth information based on grayscale images acquired by a structured light module. Such scenarios include biometric recognition scenarios (e.g., facial recognition), 3D ranging scenarios, 3D imaging, etc.

[0069] Understandably, in depth calculation scenarios, the structured light module of an electronic device can pre-control the transmitting device to project an infrared speckle field onto the object surface, and then control the receiving device to acquire the second grayscale image obtained by reflection from the target object surface; at the same time, based on the offset mapping relationship, the target offset of the calibration reference image can be determined, and the calibration reference image is corrected by the target offset to obtain the corrected calibration reference image. Then, the depth information of each pixel in the second grayscale image is calculated by combining the corrected calibration reference image, that is, the pixel depth calculation processing of the second grayscale image is performed.

[0070] Indicatively, an electronic device can use a second grayscale image (speckle grayscale image) and a corrected calibration reference image, combined with relevant calculation parameters such as the internal / external parameters of the structured light module, to calculate the depth information of each pixel on the (speckle) grayscale image-IR image through devices such as chips and application-specific integrated circuits, thereby outputting a depth map after depth calculation processing.

[0071] For example, such as Figure 3As shown, it is a schematic diagram of a depth computing application scenario involved in the embodiment of this specification. The depth computing application scenario is a face recognition scenario, including an electronic device 10, a service platform 20, a target object 30, and a camera 40 externally connected or built into the electronic device 10; wherein, the camera 40 may include at least a structured light module.

[0072] The structured light module emits a structured light beam onto the target object 30 in the shooting environment. The camera 40, controlled by the structured light module, captures the shooting environment to obtain a second grayscale image to be processed. The electronic device converts the captured second grayscale image into a depth map based on a pre-stored calibration reference image through the structured light module.

[0073] In one optional implementation, the camera 40 can also capture images of the target object 30 in the shooting environment to obtain an RGB image containing the target object 30 and the aforementioned second grayscale image. After converting the second grayscale image, depth calculation is performed to obtain a depth map. The RGB image is then sent to the service platform 10 for facial recognition.

[0074] Optionally, the electronic device 10 can also detect whether the RGB image contains a face. If so, it outputs the face bounding box position and the coordinates of the facial key points. If there are multiple RGB images, it outputs the face position and the coordinates of the facial key points in each RGB image.

[0075] The process of converting the second grayscale image and then performing depth calculation to obtain a depth map can be as follows: After obtaining the second grayscale image, the target offset relative to the calibration reference image can be determined based on the offset mapping relationship. The calibration reference image is then corrected by the target offset to obtain the corrected calibration reference image. Finally, the depth information of each pixel in the second grayscale image is calculated by combining the corrected calibration reference image. In other words, the second grayscale image is processed by pixel depth calculation to output a depth map.

[0076] In illustrative terms, electronic device 10 may be equipped with a facial recognition application. Through the facial recognition application, facial recognition can be performed on electronic device 10 itself or on service platform 20 using a facial recognition algorithm based on RGB images, depth maps, and facial information (facial position and coordinates of facial key points).

[0077] To illustrate, taking the aforementioned offset mapping relationship A as an example, this offset mapping relationship A is typically a mapping between a calibration reference image and an image offset. It can be understood that the calibration reference image usually corresponds to an image offset in offset mapping relationship A. That is, the electronic device can directly obtain the target offset for the calibration reference image based on the offset mapping relationship.

[0078] To illustrate, considering the varying impacts of different environmental conditions on depth calculation based on the calibration reference image, taking the aforementioned offset mapping relationship B as an example, the structured light module of the electronic device determines the current target environmental condition parameters (such as the current temperature parameter) while acquiring the second grayscale image. These target environmental condition parameters include, but are not limited to, fitting one or more parameters such as target time, target scene, target temperature, and target humidity. Then, the target offset corresponding to the target environmental condition parameter is searched in offset mapping relationship B. It is understandable that different target environmental condition parameters will result in different determined target offsets. For example, if the target environmental condition parameter is the target temperature parameter, then the target offset corresponding to the target temperature parameter is searched in offset mapping relationship B.

[0079] In one or more embodiments of this specification, by acquiring a first grayscale image and a calibration reference image based on the structured light module, and determining the offset mapping relationship with respect to the calibration reference image based on the first grayscale image, in depth calculation scenarios, the target offset amount relative to the calibration reference image can be determined based on the offset mapping relationship, thereby performing depth correction processing on the structured light module. This can significantly reduce or even eliminate the impact of uncertainties such as vibration, deformation, and temperature on depth calculation scenarios. Furthermore, in scenarios where the calibration reference image is difficult to modify, the target offset amount of the calibration reference image determined based on the offset mapping relationship can overcome large offsets in the calibration reference image, improving the offset robustness in depth calculation scenarios and ensuring depth calculation quality. Moreover, even when uncertainties cause large offsets in the calibration reference image and the user does not have the ability to update the calibration reference image, the depth calculation accuracy of the structured light module can still be guaranteed, ensuring its shooting performance.

[0080] Please see Figure 4 , Figure 4 This is a schematic flowchart of another embodiment of an image processing method proposed in one or more embodiments of this specification. Specifically:

[0081] S202: Obtain a first grayscale image of the structured light module and a calibration reference image corresponding to the structured light module;

[0082] The calibration reference image is a grayscale image generated by the structured light module at a calibration distance from the structured light module;

[0083] For details, please refer to one or more embodiments of this specification for the method steps, which will not be repeated here.

[0084] Furthermore, after acquiring the first grayscale image and the calibration reference image, the offset mapping relationship for the calibration reference image can be determined based on the first grayscale image.

[0085] S204: Perform at least one round of image offset processing on the calibration reference image to obtain an offset calibration image after each round of image offset processing;

[0086] The offset calibration image can be understood as a grayscale image obtained by performing at least one round of image offset processing on a calibration reference image according to a certain offset amount. It is understood that in each round of image offset processing, the offset amount in each round can be the same or different.

[0087] In one or more embodiments of this specification, the electronic device can perform round-by-round offsetting on the calibration reference image to obtain the offset calibration image after each round of image offsetting processing and record the offset amount after each round of offsetting. After each round of offsetting, the offset mapping relationship for the reference calibration image is determined based on the image correlation of each round of offsetting by measuring the image correlation between the offset calibration image after each round of offsetting and the first grayscale image.

[0088] The offset can be a custom offset, or it can be a dynamically adjusted offset for the next round of image offset processing based on the image correlation of the current round. For example, if the image correlation of the current round indicates that the correlation is weak, the next round of image offset processing can perform image offset with a larger offset.

[0089] In one feasible implementation, in order to achieve accurate quantization and improve the accuracy of offset correction during offset calculation, the offset object can be further refined instead of using the entire grayscale image as the offset object. Several refined image regions (also called image blocks) in the grayscale image are selected. Taking the image regions of the grayscale image as objects, the image correlation between the corresponding image regions (also called image blocks) between the calibration reference image and the offset calibration image after image offset processing of the first grayscale image is calculated. Based on the image correlation of each round, the offset mapping relationship of the corresponding image regions (also called image blocks) for the reference calibration image is determined.

[0090] Indicatively, the electronic device performs at least one round of image offset processing on the calibration reference image to obtain an offset calibration image after each round of image offset processing. Specifically, this can be:

[0091] A2. The electronic device determines at least one reference calibration region for the calibration reference image in each round and the reference offset corresponding to the reference calibration region;

[0092] The reference calibration region (also known as the reference calibration block) is a grayscale region in the calibration reference image. By selecting several reference calibration regions of the calibration reference image as the offset processing objects, the accuracy of offset correction can be further improved by precise quantization. In some embodiments, interference regions in the entire grayscale image can also be filtered out at the same time.

[0093] Optionally, the reference calibration region can be selected by performing key object recognition on the first color image corresponding to the first grayscale image, and determining the region where the key object is located as the reference calibration region. For example, taking a grayscale image of a face, the region where the key object is located is usually the region where the key points of the face are located, such as the left eye region, the right eye region, the left corner of the mouth region, the right corner of the mouth region, etc.

[0094] Specifically, a first color image corresponding to the first grayscale image can be obtained. The image acquisition objects of the first grayscale image and the first color image are the same object. By performing image object region detection on the first color image, at least one reference color region is obtained. The reference grayscale region of the reference color region in the first grayscale image is obtained. In the calibration reference image, the reference calibration region corresponding to each round of reference grayscale region is determined.

[0095] The first color image is an RGB color image, which can be understood as the electronic device acquiring the first color image of the same object simultaneously when the structured light module of the electronic device acquires the first grayscale image of the same object. The first color image corresponds to the first grayscale image.

[0096] Specifically, by performing key object recognition on the first color image, the region where the key object is located is determined as a reference color region, and then the reference gray region corresponding to each of the reference color regions in the first grayscale image is obtained.

[0097] As an example, the selection of key objects is customized based on the specific depth calculation scenario. For example, if the depth calculation scenario is 3D face detection, the selection of key objects can be key facial regions, regions composed of key facial points, etc.

[0098] Optionally, the selection of the reference calibration region can be a custom division of the point of interest (POI) region between the calibration reference image and / or the first grayscale image.

[0099] Optionally, image brightness detection can be performed on images (such as calibration reference images, first grayscale images, and first color images) to filter out and mark bright, dark, and background areas. The selection of the reference calibration region involves filtering out excessively dark, excessively bright, and background areas. For example, in a grayscale facial image, excessively dark areas could be the eye area, while excessively bright areas could be the eyeglass frames or the tip of the nose. This can be achieved by combining the IR image characteristics (or binarized images) from the calibration reference images, first grayscale images, and first color images, the binarization features of the grayscale images after IR image binarization, and facial keypoint coordinates. For example, for excessively bright areas, large areas of all 1s can be designated as non-reference calibration regions; for excessively dark areas, large areas of all 0s can be designated as non-reference calibration regions.

[0100] A4. Based on the reference offset, perform image offset on each of the reference calibration regions of the calibration reference image to obtain at least one offset image region after each round of image offset processing and an offset calibration image containing the at least one offset image region.

[0101] The offset image region (also known as the offset image block) is the image region obtained by offsetting the image based on the reference calibration region of the calibration reference image with the reference offset amount of the current round.

[0102] Understandably, the process of image offsetting each reference calibration region to obtain an offset image region can be independent and non-interfering. That is, image offsetting of each reference calibration region can be performed synchronously or asynchronously. After each round of image offsetting processing of the reference calibration region of the calibration reference image to obtain at least one offset image region, an offset calibration image containing at least one offset image region can be obtained for the original calibration reference image.

[0103] Schematic, the reference offset can consist of an offset adjustment value and an offset adjustment direction. The electronic device performs image offsetting on each of the reference calibration regions of the calibration reference image based on the reference offset, specifically:

[0104] During each round of image offsetting, the electronic device first determines the offset adjustment value and offset adjustment direction corresponding to the reference offset, and then uses the offset adjustment value to offset the image of each reference calibration area along the offset adjustment direction. For example, if the offset adjustment direction is vertical, the offset adjustment value can be i offset units (such as i offset rows, where i is an integer).

[0105] Schematic, the offset adjustment direction can be represented by a direction angle;

[0106] Indicatively, the offset adjustment value can be a numerical value indicating an offset of i units in the opposite direction of the offset adjustment, where i is an integer.

[0107] According to some embodiments, the reference offset can be a custom offset, such as being adjusted in each round according to a custom offset adjustment value and offset adjustment direction; or it can be dynamically adjusted in combination with the image correlation of the current round to adjust the size of the reference offset for the next round of image offset processing. For example, if the image correlation of the current round indicates that the correlation is weak, the next round of image offset processing can perform image offset with a larger reference offset.

[0108] S206: Based on the offset calibration image and the first grayscale image in each round, determine the offset mapping relationship for the calibration reference image.

[0109] The correlation detection process can be understood as a measure of the correlation between two images (the overall image and / or an image region). The correlation detection process can obtain the image correlation degree (also known as the image correlation coefficient) between the two images.

[0110] Indicatively, the electronic device, in performing the correlation detection processing based on the offset calibration image and the first grayscale image in each round, obtains the offset mapping relationship for the calibration reference image after correlation detection processing. Specifically, this can be:

[0111] B2: Determine the image correlation between the offset calibration image and the first grayscale image in each round;

[0112] In one or more embodiments of this specification, after image offsetting is performed on each reference calibration region to obtain an offset image region, image region correlation detection can be performed on the reference grayscale region in the first grayscale image and its offset image region in the offset calibration image to construct an offset mapping relationship with image region as a fine-grained dimension.

[0113] Indicatively, the electronic device performs the determination of the image correlation between the offset calibration image and the first grayscale image in each round, specifically:

[0114] The electronic device can acquire at least one offset image region corresponding to each round of offset calibration image, and determine the reference grayscale region corresponding to the first grayscale image. This can be understood as: the offset image region obtained by accumulating image offsets on the reference calibration region of the calibration reference image based on the reference offset amount i; if there are multiple reference calibration regions of the calibration reference image, then each round will correspond to multiple offset image regions in the offset calibration image; the offset image region of any round is a grayscale image region obtained by accumulating reference offset amounts i for several rounds based on the initial reference calibration region; after determining the several offset image regions corresponding to each round of offset calibration image, the reference grayscale region corresponding to the first grayscale image can be acquired accordingly. Since the reference grayscale region in the first grayscale image does not undergo image offset, after determining the reference grayscale region during the first round of image offset, it will not change subsequently and can be directly acquired by the electronic device.

[0115] After determining the offset image region and the reference grayscale region corresponding to the first grayscale image, the electronic device calculates the regional image correlation between the offset image region and the reference grayscale region, and determines the image correlation based on the regional image correlation in each round. It can be understood that, assuming there are x offset image regions, there are usually also x reference grayscale regions corresponding to the first grayscale image. For each offset image region, the regional image correlation between the offset image region and the reference grayscale region is calculated, typically resulting in x different regional image correlations. Assuming the image offset is y rounds, then for the same set of "offset image regions and reference grayscale regions", there will be y region image correlations. Typically, there are x sets of "offset image regions and reference grayscale regions" corresponding to x*y region image correlations. For the same region object, a target correlation can be determined based on the y region image correlations (e.g., the target correlation indicated by the maximum correlation). For the same region object corresponding to a set of "offset image regions and reference grayscale regions", the target correlation is also the image correlation of the region object included in the reference.

[0116] Optionally, the correlation of the regions obtained from image region correlation detection is assumed to be represented by Corr-A, and the correlation detection process can be performed using the following correlation calculation formula:

[0117] Corr-A(i,j)=I Obj (i,j)*I Ref (i,j)

[0118] Where i represents the reference offset, j represents the image offset in the j-th round, and I Obj (i,j) represents the reference grayscale region of the first grayscale image during the j-th round of image offset, IRef (i,j) represents the offset image region of the offset calibration image obtained after accumulating the image offset of the calibration reference image based on the reference offset amount i during the j-th round of image offset. Obj (i,j)*I Ref "(i,j)" represents the calculation of the regional image correlation between the reference gray area and the offset image area - Corr-A(i,j).

[0119] In one or more embodiments of this specification, the image correlation between the two images in each round can be obtained by performing overall image correlation detection based on the offset calibration image and the first grayscale image in each round, so as to construct an offset mapping relationship with the overall image as a fine-grained dimension based on the image correlation of each round.

[0120] In illustrative terms, the image correlation obtained from overall image correlation detection is assumed to be represented by Corr-B. The correlation detection process can be performed using the following correlation calculation formula:

[0121] Corr-B(i,j)=I Obj (i,j)*I Ref (i,j)

[0122] Where i represents the reference offset, j represents the image offset in the j-th round, and I Obj (i,j) represents the first grayscale image in the j-th round of image shifting, I Ref (i,j) represents the offset calibration image obtained after accumulating the image offset of the calibration reference image based on the reference offset amount i during the j-th round of image offset. Obj (i,j)*I Ref "(i,j)" represents the image correlation between the offset calibration image and the first grayscale image -Corr-B(i,j).

[0123] Optionally, in one or more embodiments of this specification, the correlation detection processing may use a correlation calculation method for related images as the measurement standard for image correlation in this specification; for example, the sum of errors (SAD) may be used as the measurement standard for image correlation, and generally, the smaller the SAD, the higher the similarity; or the correlation detection processing may use covariance as the measurement standard for image correlation, and the closer its value is to 1, the closer the linear correlation between the two images, and the closer its value is to 0, the less close the linear correlation between the two images, and so on.

[0124] B4: Determine the target correlation from each of the image correlations, and obtain the image offset relative to the calibration reference image indicated by the target correlation;

[0125] In one or more embodiments of this specification, when there are multiple image relevances (or region image relevances), the multiple image relevances can be aggregated (selected) to obtain the aggregated (selected) target relevance, which is then incorporated into the reference for establishing the relevance mapping relationship A. One approach is to take the average relevance of the multiple image relevances as the image relevance; another approach is to take the mode of the multiple image relevances as the image relevance; yet another approach is to take the median of the multiple image relevances as the image relevance, and so on.

[0126] Schematic, determining the target relevance from the image relevances can specifically be:

[0127] The maximum relevance is obtained from all the image relevance scores, and this maximum relevance score is used as the target relevance score. For example, if the image relevance score is the regional image relevance score for a specific regional object, then for the same regional object, the maximum relevance score can be obtained from all the regional image relevance scores, and this maximum relevance score can be used as the target relevance score for the regional object.

[0128] Understandably, after determining the target correlation from each of the image correlations, the electronic device can obtain the image offset relative to the calibration reference image indicated by the target correlation.

[0129] To illustrate, taking the target relevance as the overall image relevance as an example, suppose the target relevance is obtained after performing image offset processing on the calibration reference image in the w-th round, and then performing correlation detection between the offset calibration image in the w-th round and the first grayscale image. Then the image offset indicated by the target relevance is the total reference offset of all w (w is an integer) rounds of image offset processing. This total reference offset can be understood as the cumulative offset of the w rounds of image offset processing, that is, the total reference offset obtained by accumulating the reference offset of each round in w rounds, and using this total reference offset as the target offset.

[0130] To illustrate, taking the target correlation as the region image correlation of a region in an image as an example, suppose the target correlation is obtained after performing image offset processing on a certain reference calibration region of the calibration reference image in the z-th round, and performing correlation detection on the offset image region of the offset calibration image in the z-th round and the corresponding reference grayscale region of the first grayscale image. Then the image offset indicated by the target correlation is the total reference offset for the reference calibration region in all z (z is an integer) rounds of image offset processing. This total reference offset can be understood as the cumulative offset of the image offset processing for the reference calibration region in z rounds, that is, the total reference offset obtained by accumulating the reference offset for the reference calibration region in each of the z rounds, and using this total reference offset as the target offset.

[0131] In a specific implementation scenario, image region correlation detection is performed on a reference grayscale region in a first grayscale image and its offset image region in an offset calibration image. The region image correlation between the offset image region and the reference grayscale region is calculated. Based on the region image correlation in each round, a target correlation is determined, and the image offset relative to the calibration reference image indicated by the target correlation is obtained. Further explanation is as follows:

[0132] For a reference gray area in the first gray image, such as a 30x20 area, and the corresponding reference calibration area in the calibration reference image, several rounds of image offset are performed on the reference calibration area to obtain the offset image area, and the correlation is calculated in this way.

[0133] Furthermore, assuming that the offset adjustment direction indicated by the reference offset in each round is vertical (i.e., vertical offset adjustment), and the offset adjustment value is i offset units, it is usually necessary to offset the reference calibration region in the calibration reference image for several rounds to obtain the offset image region for each round. The correlation between the offset image region and its corresponding reference grayscale region in the first grayscale image is calculated, and the most ideal correlation is selected as the target correlation from the correlation results of several rounds of regional images.

[0134] The formula for calculating the correlation between regional images is as follows:

[0135]

[0136] Where Corr() represents the region image relevance, row and col are the row and column pixel numbers, S is a candidate region, Obj is the first grayscale image, Obj(row,col) indicates that the row and column pixel numbers of the first grayscale image belong to the reference grayscale region S of the first grayscale image, Ref is the calibration reference image, Ref(row,col) indicates that the row and column pixel numbers of the calibration reference image belong to the reference grayscale region S of the calibration reference image; R represents the horizontal search range, which is usually determined based on the horizontal region parameters of the candidate region S, and i is the offset adjustment value indicated by the offset.

[0137] Based on practical applications, the range of i is [-N, N], where N represents the maximum offset adjustment range calculated, such as [-10, 10].

[0138] Indicatively, by using the above formula for calculating the correlation of regional images, R Corr reference values ​​can be obtained in the process of calculating the correlation of regional images in one round of image offset calculation. The correlation of regional images in one round is the maximum value of the correlation among the R Corr reference values. The maximum value, namely Corr(), represents the correlation of regional images.

[0139] For the same region object s, in each round, image offset adjustment is performed based on the offset adjustment value to calculate the region image correlation. Assuming the number of image offset rounds is y, then for the same group of "offset image region s and reference grayscale region s", there will be y region image correlations Corr(). A target correlation can be determined based on the y region image correlations (such as taking the target correlation indicated by the maximum correlation). The target correlation is also the image correlation of the region object (i.e., the reference calibration region) included in the reference.

[0140] Optionally, the final image offset can be determined based on the most ideal i value selected based on the correlation, or the final image offset can be determined based on the most ideal i value selected based on the correlation and the corresponding r value.

[0141] For example, R = 192 represents the range of disparity to be searched, S represents the region block in Obj to be searched for disparity, that is, the reference grayscale region s, and Ref needs to move this block from left to right R times, calculating a Corr reference value each time, so that R Corr reference values ​​can be obtained. Corr(i,S) can be the image correlation of recording the maximum value among these R Corr reference values ​​as the reference, and i is the offset adjustment value, which can be understood as the number of rows to shift the region block S (that is, the offset image region) in Ref upward or downward.

[0142] It should be noted that if the number of reference gray areas in the first gray-scale image is 'a', then 'a' target relevance scores will be obtained; furthermore, if the number of first gray-scale images is 'b' and the number of reference gray areas in the first gray-scale image is 'a', then 'a*b' target relevance scores will be obtained; both 'a' and 'b' are positive integers.

[0143] Furthermore, the image offset relative to the calibration reference image for obtaining the target relevance indicator can be expressed by the following calculation formula:

[0144] shift(S)=argMax{Corr(i,S)},i={0,±1,±2,…,±N}

[0145] Here, shift(S) represents the image offset, and Corr() represents the region image correlation. By taking the maximum correlation among several region image correlations as the target correlation, i.e., Max{Corr(i,S)}, shift(S) finds the ideal offset adjustment value i and the ideal lateral cumulative value r corresponding to the largest target correlation in Corr(i,S). At least the image offset can be obtained based on the ideal offset adjustment value i. If the image offset indicates the ideal offset adjustment value i, then the offset adjustment direction is the vertical offset direction. It can be understood that in some depth calculation scenarios (such as 3D face recognition scenarios), the focus can be on the vertical offset, while the actual lateral offset usually has a small impact and can be ignored.

[0146] Optionally, the ideal offset adjustment value i is usually the vertical offset value. When calculating Corr(i,S), there will also be an ideal horizontal cumulative value r. The ideal horizontal cumulative value r can be used as the ideal horizontal offset value. Based on the selected i, i.e. r, the image offset can be obtained.

[0147] B6: Determine the offset mapping relationship for the calibration reference image based on the image offset.

[0148] In one feasible implementation, a first offset mapping relationship between the image offset and the reference calibration area can be directly constructed;

[0149] To illustrate, taking the image offset as the regional image offset of several reference calibration regions (also called image blocks) of the calibration reference image as an example, each of the several reference calibration regions of the calibration reference image will correspond to a regional image offset. In this case, the image offset is composed of the regional image offsets of several reference calibration regions. By establishing a mapping between this regional image offset and the corresponding image region (or image block) in the calibration reference image, this mapping is also known as the first offset mapping relationship. This first offset mapping relationship is usually a mapping between several image regions (or image blocks) of the calibration reference image and the regional image offsets. It can be understood that the calibration reference image usually corresponds to multiple regional image offsets in the first offset mapping relationship.

[0150] Optionally, in determining the first offset mapping relationship: In cases where there are multiple region image offsets for the calibration reference image determined based on the same image region from multiple first grayscale images, the electronic device can aggregate these multiple region image offsets to obtain a final aggregated region image offset, which is then incorporated into the reference for establishing the first offset mapping relationship. The aggregation process can involve taking the average offset of the multiple region image offsets as the final offset, taking the mode of the multiple region image offsets as the final offset, taking the median of the multiple region image offsets as the final offset, and so on.

[0151] In one feasible implementation, considering that different environmental conditions will have different effects on depth calculation based on the calibration reference image, the different environmental conditions can be further quantified. A second offset mapping relationship for the calibration reference image can be further constructed using the first grayscale image generated by the structured light module under different environmental conditions. Specifically, reference environmental condition parameters corresponding to the first grayscale image can be obtained, and a second offset mapping relationship can be constructed between the reference environmental condition parameters, the image offset, and the reference calibration area.

[0152] Furthermore, the step of obtaining the reference environmental condition parameters corresponding to the first grayscale image and constructing a second offset mapping relationship between the reference environmental condition parameters, the image offset, and the reference calibration region can be:

[0153] D2: Determine the reference calibration area corresponding to the image offset and obtain the reference environmental condition parameters corresponding to each of the first grayscale images;

[0154] The image offset is usually determined based on the correlation between several offset image regions and the reference grayscale region; based on this, the original offset image region or the reference grayscale region in the first grayscale image can be determined in reverse according to the image offset.

[0155] One approach is to determine the initial reference calibration region corresponding to the original offset image region in the reference calibration image;

[0156] One approach is to determine the reference calibration region corresponding to the reference grayscale region in the reference calibration image.

[0157] Environmental condition parameters (such as reference environmental condition parameters) are characterized by one or more of the following: temperature, humidity, time, and scene parameters. Different combinations of these related environmental condition parameters constitute different environmental condition types.

[0158] For illustrative purposes, scene parameters can be home scene parameters, outdoor scene parameters, office scene parameters, nighttime scene parameters, etc. Time parameters can be several time ranges divided based on actual application conditions;

[0159] Understandably, when the structured light module generates several first grayscale images, the electronic device can record the corresponding reference environmental condition parameters, such as reference temperature parameters, reference time parameters, reference humidity parameters, etc., and associate the first grayscale images with the corresponding environmental conditions at that time. This makes it easier to obtain the reference environmental condition parameters corresponding to the first grayscale images later.

[0160] D4: Use the at least one reference calibration region, the image offset of the reference calibration region, and the reference environmental condition parameters as offset mapping data for each of the first grayscale images;

[0161] Understandably, the offset mapping data is determined based on the first grayscale image and the calibration reference image. Typically, the image offset, the reference calibration area of ​​the image offset in the calibration reference image, and the reference environmental condition parameters corresponding to the first grayscale image are correlated to obtain the correlated data, which is then used as the offset mapping data generated based on the first grayscale image.

[0162] Understandably, there are usually multiple first grayscale images. Based on multiple first grayscale images, multiple offset mapping data can be obtained in the manner described above. That is, each first grayscale image can correspond to an offset mapping data. The offset mapping data can be expressed in the form of "block number + reference environmental condition parameter + (region) image offset". The block number is the image region identifier corresponding to the reference calibration region.

[0163] D5: Based on the offset mapping data, construct a second offset mapping relationship between the reference environmental condition parameters, the image offset, and the reference calibration area.

[0164] In practical applications, there may be multiple offsets for the same reference calibration area among multiple offset mapping data.

[0165] The electronic device can obtain at least one first image offset corresponding to the same reference calibration region from each of the offset mapping data; then it can perform fusion processing on each of the first image offsets to obtain a second image offset for the reference calibration region; and then construct a second offset mapping relationship corresponding to the second image offset, the reference calibration region, and the reference environmental condition parameters.

[0166] To illustrate, fusing the offsets of each of the first image regions to obtain the second image offset for the reference calibration region can be achieved by fusing the offsets of multiple regions within the same reference calibration region, as follows:

[0167] Electronic devices can fuse multiple first image offsets to obtain a final image offset (i.e., a second image offset) that is then incorporated into the reference for establishing a second offset mapping relationship. The fusion process can involve taking the average offset of the multiple first image offsets as the final offset of the reference calibration region, taking the mode of the multiple first image offsets as the second image offset of the reference calibration region, taking the median of the multiple first image offsets as the second image offset of the reference calibration region, and so on.

[0168] Understandably, by determining the final second image offset using multiple offset mapping data of the same reference calibration area, the offset of multiple offset mapping data is updated. When establishing the second offset mapping relationship, the same reference calibration area is associated with its reference environmental condition parameters and the final second image offset in multiple offset mapping data, thereby obtaining the second offset mapping relationship for the reference calibration area. In the second offset mapping relationship, for a certain reference calibration area, there may be several second image offsets under different reference environmental condition parameters (such as temperature parameters).

[0169] Understandably, this allows the establishment of a second offset mapping relationship between image offsets corresponding to several reference calibration regions under several different environmental condition parameters. This second offset mapping relationship is essentially a mapping between different reference environmental condition parameters and second image offsets. Let's denote it as the second offset mapping relationship. This second offset mapping relationship is typically a mapping of the second image offsets for several reference calibration regions in the calibration reference image. The second offset mapping relationship consists of different environmental condition parameters and their corresponding second image offsets for the reference calibration regions.

[0170] Schematic, using the second offset mapping relationship as a mapping array, the corresponding mapping array can be represented as (block number, environmental condition parameters, (region) image offset). The block number is the image region identifier corresponding to the reference calibration region, and the environmental condition parameters can be, for example, temperature parameters. In subsequent depth calculation scenarios, based on the second offset mapping relationship, the current environmental condition parameters of the second grayscale image acquired by the current structured light module can be determined directly. Then, the (region) image offsets of all reference calibration regions indicated by the block numbers can be directly indexed from the second offset mapping relationship according to the current environmental condition parameters, for further depth correction processing of the depth calculation process involving the structured light module.

[0171] In a specific implementation scenario, the second offset mapping relationship is used as the mapping list, and the environmental condition parameter is used as the temperature parameter for further interpretation, as follows:

[0172] Each offset mapping data determined based on the first grayscale image can be represented in the form of "block number + reference ambient temperature parameter + (region) image offset", where the block number is the image region identifier corresponding to the reference calibration region.

[0173] When there are multiple first grayscale images, the offset mapping data is aggregated together. First, data with the same block number and temperature are merged. The average of the first image offsets of these data is calculated, and obvious first image offsets are removed. The average is used as the second image offset after merging. At this point, in the offset mapping data, the image offsets corresponding to the same block number and temperature (region) are all the same second image offset. Then, a mapping list as shown in Table 1 can be formed as follows:

[0174] Table 1

[0175]

[0176] In Table 1, the column elements in the first column represent the block number, and the row elements in the first row represent the temperature parameters. Based on the temperature, the target offset of any calibration reference area can be found in the offset mapping list corresponding to Table 1.

[0177] In some embodiments, for missing items in the offset mapping list, the missing items can be smoothly filled in according to the offset corresponding to similar temperatures and / or similar blocks.

[0178] In one feasible implementation, the offset mapping data obtained further based on different first grayscale images can be represented in the form of "block number + reference environmental condition parameter + (region) image offset". The block number is the image region identifier corresponding to the reference calibration region. The offset mapping data can be vectorized using feature engineering, that is, the offset mapping feature vectors in several offset mapping data are extracted, and then these offset mapping feature vectors are mapped into a vector feature space. By performing vector feature clustering on the offset mapping feature vectors corresponding to all offset mapping data in the vector feature space, several cluster centers can be obtained. A cluster center vector can be determined based on each cluster center. The cluster center vector is also the reference for finally incorporating the second offset mapping relationship. Each cluster center vector can be parsed to obtain cluster offset data, which can be represented as "cluster block number + cluster reference environmental condition parameter + cluster (region) image offset". These cluster offset data form the second offset mapping relationship. By using vector clustering, the commonalities of the cluster offset data can be deeply mined and interfering offset data can be filtered out, so as to accurately determine the offset mapping relationship.

[0179] S208: In the depth calculation scenario, the target offset for the calibration reference image is determined based on the offset mapping relationship, and the structured light module is subjected to depth correction processing based on the target offset.

[0180] Understandably, in depth calculation scenarios, the second grayscale image of the target object is obtained through the structured light module. Based on the offset mapping relationship and the second grayscale image, the target offset relative to the calibration reference image is determined. Then, the second grayscale image is subjected to pixel depth correction processing based on the target offset and the calibration reference image.

[0181] In a depth calculation scenario, the structured light module of an electronic device can pre-control the transmitting device to project structured light, such as infrared speckle field, onto the surface of an object, and then control the receiving device to acquire the second grayscale image obtained by reflection from the surface of the target object; and at the same time, the target offset relative to the calibration reference image can be determined based on the second offset mapping relationship.

[0182] Optionally, the electronic device may simultaneously acquire the second grayscale image and determine the target environmental condition parameters (such as target temperature) of the structured light module when generating the second grayscale image; and acquire at least one reference calibration region of the reference calibration image;

[0183] Based on the target environmental condition parameters (such as target temperature), the target area offset corresponding to each reference calibration area is determined in the second offset mapping relationship; in specific implementation, if there are multiple reference calibration areas, then multiple reference calibration areas correspond to multiple target area offsets.

[0184] A method for performing pixel depth correction processing on the second grayscale image based on the target offset and the calibration reference image may be as follows: using the target region offset to perform offset correction processing on the reference calibration region of the calibration reference image to obtain the offset-corrected calibration reference image, and performing pixel depth calculation processing on the second grayscale image based on the calibration reference image.

[0185] Indicatively, an electronic device can use the corrected second grayscale image and the calibration reference image, combined with relevant calculation parameters such as the internal / external parameters of the structured light module, to calculate the depth information of each pixel on the (speckle) grayscale-IR image through devices such as chips and application-specific integrated circuits, thereby outputting a depth map after depth calculation processing.

[0186] A method for performing pixel depth correction processing on the second grayscale image based on the target offset and the calibration reference image may be as follows: determining the target grayscale region corresponding to the reference calibration region in the second grayscale image, performing offset correction processing on the corresponding target grayscale region in the second grayscale image using each target region offset, obtaining the offset-corrected second grayscale image, and performing pixel depth calculation processing on the second grayscale image based on the calibration reference image.

[0187] Indicatively, the electronic device can use the second grayscale image and the corrected calibration reference image, combined with relevant calculation parameters such as the internal / external parameters of the structured light module, to calculate the depth information of each pixel on the (speckle) grayscale-IR image through devices such as chips and application-specific integrated circuits, thereby outputting a depth map after depth calculation processing.

[0188] In one or more embodiments of this specification, after acquiring a first grayscale image of a structured light module and a calibration reference image corresponding to the structured light module, the electronic device may further: perform image desensitization processing on the first grayscale image to obtain a first grayscale image after image desensitization processing; and perform image desensitization processing on the calibration reference image to obtain a calibration reference image after image desensitization processing.

[0189] It is understandable that in some deep computing scenarios (such as facial recognition scenarios), the calibration reference image and the first grayscale image may contain private data. Based on this, the first grayscale image and the calibration reference image can be desensitized to avoid privacy leakage.

[0190] Optionally, an image desensitization model based on a neural network can be used to desensitize the first grayscale image and the calibration reference image, and output the desensitized first grayscale image and calibration reference image.

[0191] Optionally, the first grayscale image and the calibration reference image can be binarized to obtain the processed first grayscale image and the calibration reference image. The processed first grayscale image and the calibration reference image are in the form of binary feature maps. Using the binary feature maps to desensitize user information does not affect the subsequent calculation and processing.

[0192] In one or more embodiments of this specification, before executing the image processing method, the electronic device may detect the device system load. If the device system load is less than the load threshold, the image processing method is then executed. This is to avoid insufficient allocation of system resources for the depth calculation task in the depth calculation scenario when the device system load reaches the load threshold, which would affect the depth calculation efficiency and prevent image processing from being stuck or blocked.

[0193] In one or more embodiments of this specification, before executing the image processing method, the electronic device can detect the depth calculation accuracy of the structured light module. When the depth calculation accuracy is less than the accuracy threshold, the depth calculation effect of the device is usually poor, such as large depth loss or inaccurate depth information in the depth map. Then the image processing method can be executed to update the offset and correct the depth calculation effect.

[0194] In one or more embodiments of this specification, the target offset for the calibration reference image can be determined based on the offset mapping relationship, and then the structured light module can be subjected to depth correction processing. This can significantly reduce or even eliminate the impact of uncertainties such as vibration, deformation, and temperature on the depth calculation scene. Furthermore, in scenarios where the calibration reference image is difficult to modify, the target offset of the calibration reference image determined based on the offset mapping relationship can overcome large offsets in the calibration reference image, improving the offset robustness in the depth calculation scene and ensuring the quality of depth calculation. Moreover, even when uncertainties cause large offsets to the calibration reference image and the user lacks the ability to update the calibration reference image, the depth calculation accuracy of the structured light module can still be guaranteed, ensuring its shooting performance. Furthermore, by incorporating environmental condition parameters such as temperature, humidity, scene, and time during the construction of the offset mapping relationship, the impact of these uncertainties on the depth calculation scene can be offset or mitigated, further improving the effect of depth offset correction and achieving offset quantification of the influence of uncertain environments.

[0195] The following will combine Figure 5 This manual provides a detailed description of the image processing apparatus provided. It should be noted that... Figure 5 The image processing apparatus shown is used to execute this application. Figures 1-4 The methods of the embodiments shown are illustrated only in the parts relevant to this specification for ease of explanation. For specific technical details not disclosed, please refer to this application. Figures 1-4 The example shown.

[0196] Please see Figure 5 This diagram illustrates the structure of the image processing apparatus described in this specification. The image processing apparatus 1 can be implemented as all or part of a user terminal through software, hardware, or a combination of both. According to some embodiments, the image processing apparatus 1 includes an image acquisition module 11, a relationship determination module 12, and a depth calculation module 13, specifically used for:

[0197] Image acquisition module 11 is used to acquire a first grayscale image collected by the structured light module and to acquire a calibration reference image corresponding to the structured light module. The calibration reference image is a grayscale image generated by the structured light module at a calibration distance from the structured light module.

[0198] The relationship determination module 12 is used to determine the offset mapping relationship for the calibration reference image based on the first grayscale image;

[0199] The depth calculation module 13 is used to determine the target offset for the calibration reference image based on the offset mapping relationship in the depth calculation scene, and to perform depth correction processing on the structured light module based on the target offset.

[0200] Optional, such as Figure 6 As shown, the relationship determination module 12 includes:

[0201] Image offset unit 121 is used to perform at least one round of image offset processing on the calibration reference image to obtain an offset calibration image after each round of image offset processing;

[0202] The correlation detection unit 122 is used to determine the offset mapping relationship for the calibration reference image based on the offset calibration image and the first grayscale image in each round.

[0203] Optionally, the image offset unit 121 is specifically used for:

[0204] Determine at least one reference calibration region for the calibration reference image in each round, and the reference offset corresponding to the reference calibration region;

[0205] Based on the reference offset, each of the reference calibration regions of the calibration reference image is image offset to obtain at least one offset image region after each round of image offset processing and an offset calibration image containing the at least one offset image region.

[0206] Optionally, the correlation detection unit 122 is specifically used for:

[0207] Obtain the first color image corresponding to the first grayscale image, wherein the image acquisition objects of the first grayscale image and the first color image are the same object;

[0208] Perform image object region detection on the first color image to obtain at least one reference color region, and obtain the reference grayscale region of the reference color region in the first grayscale image.

[0209] In the calibration reference image, a reference calibration region is determined for each round of the reference grayscale region.

[0210] Optionally, the image offset unit 121 is specifically used for:

[0211] Determine the offset adjustment value and offset adjustment direction corresponding to the reference offset, and perform image offset on each of the reference calibration regions along the offset adjustment direction using the offset adjustment value.

[0212] Optional, such as Figure 7 As shown, the correlation detection unit 122 includes:

[0213] The correlation determination subunit 1221 is used to determine the image correlation between the offset calibration image and the first grayscale image in each round;

[0214] Offset acquisition subunit 1222 is used to determine the target correlation from each of the image correlations and acquire the image offset relative to the calibration reference image indicated by the target correlation;

[0215] The relationship determination subunit 1223 is used to determine the offset mapping relationship for the calibration reference image based on the image offset.

[0216] Optionally, the relevance determination subunit 1221 is specifically used for:

[0217] Obtain at least one offset image region corresponding to the offset calibration image in each round, and determine that the offset image region is in the reference grayscale region corresponding to the first grayscale image;

[0218] Calculate the regional image correlation between the offset image region and the reference grayscale region, and use the regional image correlation in each round as the image correlation.

[0219] Optionally, the offset acquisition subunit 1222 is used for:

[0220] The maximum correlation score is obtained from all the image correlation scores, and the maximum correlation score is used as the target correlation score.

[0221] Optionally, the relationship-determining subunit 1223 is used for:

[0222] Construct a first offset mapping relationship between the image offset and the reference calibration region; and / or,

[0223] Obtain the reference environmental condition parameters corresponding to the first grayscale image, and construct a second offset mapping relationship between the reference environmental condition parameters, the image offset, and the reference calibration area.

[0224] Optionally, the relationship-determining subunit 1223 is used for:

[0225] Determine the reference calibration area corresponding to the image offset, and obtain the reference environmental condition parameters corresponding to the first grayscale image;

[0226] The at least one reference calibration region, the image offset of the reference calibration region, and the reference environmental condition parameters are used as offset mapping data for each of the first grayscale images;

[0227] Based on the offset mapping data, a second offset mapping relationship is constructed between the reference environmental condition parameters, the image offset, and the reference calibration area.

[0228] Optionally, the relationship-determining subunit 1223 is used for:

[0229] In each of the offset mapping data, at least one first image offset corresponding to the same reference calibration region is obtained;

[0230] The first image offsets are fused according to the reference environmental condition parameters to obtain the second image offset for the reference calibration area.

[0231] Construct a second offset mapping relationship corresponding to the second image offset, the reference calibration region, and the reference environmental condition parameters.

[0232] Optionally, the device 1 is further configured to:

[0233] Perform image desensitization processing on the first grayscale image to obtain the first grayscale image after image desensitization processing; and / or,

[0234] The calibration reference image is subjected to image desensitization processing to obtain the calibration reference image after image desensitization processing.

[0235] Optionally, the depth calculation module 13 is specifically used for:

[0236] The structured light module is used to obtain the second grayscale image of the target object.

[0237] Based on the offset mapping relationship and the second grayscale image, the target offset for the calibration reference image is determined;

[0238] The second grayscale image is subjected to pixel depth correction processing based on the target offset and the calibration reference image.

[0239] Optionally, the depth calculation module 13 is specifically used for:

[0240] Obtain the target environmental condition parameters when generating the second grayscale image, and obtain at least one reference calibration region of the reference calibration image;

[0241] Based on the target environmental condition parameters, the target region offset corresponding to each of the reference calibration regions is determined in the offset mapping relationship;

[0242] The pixel depth correction processing of the second grayscale image based on the target offset and the calibration reference image includes:

[0243] The target region offset is used to perform offset correction processing on the reference calibration region of the calibration reference image to obtain the offset-corrected calibration reference image. Pixel depth calculation processing is then performed on the second grayscale image based on the calibration reference image; or,

[0244] The target grayscale region corresponding to the reference calibration region in the second grayscale image is determined, and the target grayscale region of the second grayscale image is offset and corrected using the target region offset to obtain the second grayscale image after offset correction. Pixel depth calculation is performed on the second grayscale image based on the calibration reference image.

[0245] Optionally, environmental condition parameters include at least one of temperature parameters, humidity parameters, scene parameters, and time parameters.

[0246] It should be noted that the image processing apparatus provided in the above embodiments is only illustrated by the division of the above functional modules when executing the image processing method. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the image processing apparatus and the image processing method embodiments provided in the above embodiments belong to the same concept, and the implementation process is detailed in the method embodiments, which will not be repeated here.

[0247] The serial numbers in this specification are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0248] In one or more embodiments of this specification, the target offset for the calibration reference image can be determined based on the offset mapping relationship, and then the structured light module can be subjected to depth correction processing. This can significantly reduce or even eliminate the impact of uncertainties such as vibration, deformation, and temperature on the depth calculation scene. Furthermore, in scenarios where the calibration reference image is difficult to modify, the target offset of the calibration reference image determined based on the offset mapping relationship can overcome large offsets in the calibration reference image, improving the offset robustness in the depth calculation scene and ensuring the quality of depth calculation. Moreover, even when uncertainties cause large offsets to the calibration reference image and the user lacks the ability to update the calibration reference image, the depth calculation accuracy of the structured light module can still be guaranteed, ensuring its shooting performance. Furthermore, by incorporating environmental condition parameters such as temperature, humidity, scene, and time during the construction of the offset mapping relationship, the impact of these uncertainties on the depth calculation scene can be offset or mitigated, further improving the effect of depth offset correction and achieving offset quantification of the influence of uncertain environments.

[0249] This specification also provides a computer storage medium capable of storing multiple instructions adapted to be loaded and executed by a processor as described above. Figures 1-4 The image processing method described in the illustrated embodiment can be found in the following document for a detailed execution process. Figures 1-4 The specific details of the illustrated embodiments will not be elaborated here.

[0250] This application also provides a computer program product storing at least one instruction, which is loaded and executed by the processor as described above. Figures 1-4 The image processing method described in the illustrated embodiment can be found in the following document for a detailed execution process. Figures 1-4 The specific details of the illustrated embodiments will not be elaborated here.

[0251] Please refer to Figure 8 This diagram illustrates a structural block diagram of an electronic device provided in an exemplary embodiment of this application. The electronic device in this application may include one or more components such as a processor 110, a memory 120, an input device 130, an output device 140, and a bus 150. The processor 110, memory 120, input device 130, and output device 140 may be connected via the bus 150.

[0252] Processor 110 may include one or more processing cores. Processor 110 connects to various parts of the electronic device via various interfaces and lines, and performs various functions and processes data of electronic device 100 by running or executing instructions, programs, code sets, or instruction sets stored in memory 120, and by calling data stored in memory 120. Optionally, processor 110 may be implemented using at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), or programmable logic array (PLA). Processor 110 may integrate one or more of the following: central processing unit (CPU), graphics processing unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also not be integrated into processor 110 and may be implemented separately through a communication chip.

[0253] The memory 120 may include random access memory (RAM) or read-only memory (ROM). Optionally, the memory 120 may include a non-transitory computer-readable storage medium. The memory 120 may be used to store instructions, programs, code, code sets, or instruction sets. The memory 120 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as touch functionality, sound playback functionality, image playback functionality, etc.), instructions for implementing the various method embodiments described below, etc. The operating system may be the Android system, including systems deeply developed based on the Android system, the iOS system developed by Apple Inc., including systems deeply developed based on the iOS system, or other systems. The data storage area may also store data created by the electronic device during use, such as phonebook data, audio and video data, chat log data, etc.

[0254] See Figure 9 As shown, the memory 120 can be divided into operating system space and user space. The operating system runs in the operating system space, while native and third-party applications run in the user space. To ensure that different third-party applications can achieve good running performance, the operating system allocates corresponding system resources for each application. However, different application scenarios within the same third-party application have different requirements for system resources. For example, in local resource loading scenarios, third-party applications have high requirements for disk read speed; in animation rendering scenarios, third-party applications have high requirements for GPU performance. Since the operating system and third-party applications are independent of each other, the operating system often cannot promptly perceive the current application scenario of a third-party application, resulting in the operating system's inability to adapt system resources accordingly to the specific application scenario of the third-party application.

[0255] In order for the operating system to distinguish the specific application scenarios of third-party applications, it is necessary to establish data communication between the third-party applications and the operating system. This would allow the operating system to obtain the current scenario information of the third-party applications at any time, and then perform targeted system resource adaptation based on the current scenario.

[0256] Taking the Android operating system as an example, the programs and data stored in memory 120 are as follows: Figure 10As shown, the memory 120 can store the Linux kernel layer 320, the system runtime library layer 340, the application framework layer 360, and the application layer 380. The Linux kernel layer 320, system runtime library layer 340, and application framework layer 360 belong to the operating system space, while the application layer 380 belongs to the user space. The Linux kernel layer 320 provides low-level drivers for various hardware components of the electronic device, such as display drivers, audio drivers, camera drivers, Bluetooth drivers, Wi-Fi drivers, and power management. The system runtime library layer 340 provides support for key features of the Android system through several C / C++ libraries. For example, the SQLite library provides database support, the OpenGL / ES library provides 3D graphics support, and the Webkit library provides browser kernel support. The system runtime library layer 340 also provides the Android runtime library, which mainly provides core libraries that allow developers to write Android applications using the Java language. The Application Framework Layer 360 provides various APIs that may be used when building applications. Developers can also use these APIs to build their own applications, such as activity management, window management, view management, notification management, content provider, package management, call management, resource management, and location management. At least one application runs in the Application Layer 380. These applications can be native applications that come with the operating system, such as contacts, SMS, clock, and camera apps; or third-party applications developed by third-party developers, such as games, instant messaging, and photo editing apps.

[0257] Taking the operating system as an example (iOS), the programs and data stored in memory 120 are as follows: Figure 11As shown, the iOS system includes: Core OS layer 420, Core Services layer 440, Media layer 460, and Cocoa Touch layer 480. Core OS layer 420 includes the operating system kernel, drivers, and low-level program frameworks. These low-level program frameworks provide hardware-level functionality for use by the program frameworks located in Core Services layer 440. Core Services layer 440 provides system services and / or program frameworks required by applications, such as Foundation framework, account framework, advertising framework, data storage framework, network connectivity framework, geolocation framework, motion framework, etc. Media layer 460 provides applications with audiovisual interfaces, such as interfaces related to graphics and images, audio technology, video technology, and AirPlay (wireless playback of audio and video transmission technologies). Cocoa Touch layer 480 provides various commonly used interface-related frameworks for application development and is responsible for user touch interaction on electronic devices. Examples include local notification services, remote push services, advertising frameworks, game tool frameworks, message user interface (UI) frameworks, UIKit user interface frameworks, map frameworks, and so on.

[0258] exist Figure 11 The framework shown includes, but is not limited to, the base framework in the core service layer 440 and the UIKit framework in the touchable layer 480. The base framework provides many basic object classes and data types, offering the most basic system services to all applications, and is independent of the UI. The UIKit framework, on the other hand, provides a basic UI class library for creating touch-based user interfaces. iOS applications can use the UIKit framework to provide their UI, thus providing the application's infrastructure for building user interfaces, drawing, handling user interaction events, responding to gestures, and so on.

[0259] The methods and principles for implementing data communication between third-party applications and the operating system in the iOS system can be referenced from the Android system, and will not be elaborated here.

[0260] The input device 130 is used to receive input instructions or data, and includes, but is not limited to, a keyboard, mouse, camera, microphone, or touch device. The output device 140 is used to output instructions or data, and includes, but is not limited to, a display device and a speaker. In one example, the input device 130 and the output device 140 can be combined into a touch screen, which is used to receive touch operations from the user using a finger, stylus, or any suitable object on or near it, and to display the user interface of various applications. The touch screen is usually located on the front panel of the electronic device. The touch screen can be designed as a full-screen, curved screen, or irregularly shaped screen. The touch screen can also be designed as a combination of a full-screen and a curved screen, or a combination of an irregularly shaped screen and a curved screen; this specification does not limit this.

[0261] In addition, those skilled in the art will understand that the structure of the electronic device shown in the above figures does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements. For example, the electronic device may also include radio frequency circuits, input units, sensors, audio circuits, wireless fidelity (WiFi) modules, power supplies, Bluetooth modules, etc., which will not be described in detail here.

[0262] In this specification, the entity executing each step can be the electronic device described above. Optionally, the entity executing each step can be the operating system of the electronic device. The operating system can be Android, iOS, or other operating systems; this specification does not limit this.

[0263] The electronic device described in this manual may also be equipped with a display device. This display device can be any device capable of displaying information, such as a cathode ray tube display (CR), a light-emitting diode display (LED), an e-ink screen, a liquid crystal display (LCD), or a plasma display panel (PDP). Users can use the display device on electronic device 101 to view displayed text, images, videos, and other information. The electronic device may be a smartphone, tablet computer, gaming device, AR (Augmented Reality) device, automobile, data storage device, audio playback device, video playback device, laptop, desktop computing device, or wearable device such as an electronic watch, electronic glasses, electronic helmet, electronic bracelet, electronic necklace, or electronic clothing.

[0264] exist Figure 8 In the illustrated electronic device, the processor 110 can be used to call the application program stored in the memory 120 and specifically perform the following operations:

[0265] Acquire a first grayscale image captured by the structured light module, and acquire a calibration reference image corresponding to the structured light module, wherein the calibration reference image is a grayscale image generated by the structured light module at a calibration distance from the structured light module;

[0266] Based on the first grayscale image, determine the offset mapping relationship for the calibration reference image;

[0267] In the depth calculation scenario, the target offset for the calibration reference image is determined based on the offset mapping relationship, and the structured light module is subjected to depth correction processing based on the target offset.

[0268] In one embodiment, when the processor 110 performs the operation of determining the offset mapping relationship for the calibration reference image based on the first grayscale image, it specifically performs the following operations:

[0269] Perform at least one round of image offset processing on the calibration reference image to obtain the offset calibration image after each round of image offset processing;

[0270] Based on the offset calibration image and the first grayscale image in each round, an offset mapping relationship is determined for the calibration reference image.

[0271] In one embodiment, when the processor 110 performs at least one round of image offset processing on the calibration reference image to obtain the offset calibration image after each round of image offset processing, it specifically performs the following operations:

[0272] Determine at least one reference calibration region for the calibration reference image in each round, and the reference offset corresponding to the reference calibration region;

[0273] Based on the reference offset, each of the reference calibration regions of the calibration reference image is image offset to obtain at least one offset image region after each round of image offset processing and an offset calibration image containing the at least one offset image region.

[0274] In one embodiment, the processor 110, in performing the determination of at least one reference calibration region for each round of the calibration reference image, includes:

[0275] Obtain the first color image corresponding to the first grayscale image, wherein the image acquisition objects of the first grayscale image and the first color image are the same object;

[0276] Perform image object region detection on the first color image to obtain at least one reference color region, and obtain the reference grayscale region of the reference color region in the first grayscale image.

[0277] In the calibration reference image, a reference calibration region is determined for each round of the reference grayscale region.

[0278] In one embodiment, the processor 110 performs image offsetting on each of the reference calibration regions of the calibration reference image based on the reference offset, including:

[0279] Determine the offset adjustment value and offset adjustment direction corresponding to the reference offset, and perform image offset on each of the reference calibration regions along the offset adjustment direction using the offset adjustment value.

[0280] In one embodiment, the processor 110, when performing the process of determining the offset mapping relationship for the calibration reference image based on the offset calibration image and the first grayscale image in each round, includes:

[0281] Determine the image correlation between the offset calibration image and the first grayscale image in each round;

[0282] Determine the target correlation from each of the image correlations, and obtain the image offset relative to the calibration reference image indicated by the target correlation;

[0283] The offset mapping relationship for the calibration reference image is determined based on the image offset.

[0284] In one embodiment, when the processor 110 performs the step of determining the image correlation between the offset calibration image and the first grayscale image in each round, it specifically executes the following steps:

[0285] Obtain at least one offset image region corresponding to the offset calibration image in each round, and determine that the offset image region is in the reference grayscale region corresponding to the first grayscale image;

[0286] Calculate the regional image correlation between the offset image region and the reference grayscale region, and use the regional image correlation in each round as the image correlation.

[0287] In one embodiment, when the processor 110 performs the step of determining the target relevance from the image relevances, it specifically executes the following steps:

[0288] The maximum correlation score is obtained from all the image correlation scores, and the maximum correlation score is used as the target correlation score.

[0289] In one embodiment, when the processor 110 performs the step of determining the offset mapping relationship for the calibration reference image based on the image offset, it specifically executes the following steps:

[0290] Construct a first offset mapping relationship between the image offset and the reference calibration region; and / or,

[0291] Obtain the reference environmental condition parameters corresponding to the first grayscale image, and construct a second offset mapping relationship between the reference environmental condition parameters, the image offset, and the reference calibration area.

[0292] In one embodiment, when there are multiple first grayscale images, the processor 110, when performing the steps of acquiring the reference environmental condition parameters corresponding to the first grayscale images and constructing the second offset mapping relationship between the reference environmental condition parameters, the image offset, and the reference calibration area, specifically executes the following steps:

[0293] Determine the reference calibration area corresponding to the image offset, and obtain the reference environmental condition parameters corresponding to the first grayscale image;

[0294] The at least one reference calibration region, the image offset of the reference calibration region, and the reference environmental condition parameters are used as offset mapping data for each of the first grayscale images;

[0295] Based on the offset mapping data, a second offset mapping relationship is constructed between the reference environmental condition parameters, the image offset, and the reference calibration area.

[0296] In one embodiment, when the processor 110 executes the process of constructing the second offset mapping relationship between the reference environmental condition parameters, the image offset, and the reference calibration region based on the offset mapping data, it specifically performs the following steps:

[0297] In each of the offset mapping data, at least one first image offset corresponding to the same reference calibration region is obtained;

[0298] The first image offsets are fused according to the reference environmental condition parameters to obtain the second image offset for the reference calibration area.

[0299] Construct a second offset mapping relationship corresponding to the second image offset, the reference calibration region, and the reference environmental condition parameters.

[0300] In one embodiment, after acquiring the first grayscale image of the structured light module and the corresponding calibration reference image of the structured light module, the processor 110 performs the following steps:

[0301] Perform image desensitization processing on the first grayscale image to obtain the first grayscale image after image desensitization processing; and / or,

[0302] The calibration reference image is subjected to image desensitization processing to obtain the calibration reference image after image desensitization processing.

[0303] In one embodiment, when the processor 110 performs the steps of determining the target offset for the calibration reference image based on the offset mapping relationship and performing depth correction processing on the structured light module based on the target offset, the processor 110 specifically executes the following steps:

[0304] The structured light module is used to obtain the second grayscale image of the target object.

[0305] Based on the offset mapping relationship and the second grayscale image, the target offset for the calibration reference image is determined;

[0306] The second grayscale image is subjected to pixel depth correction processing based on the target offset and the calibration reference image.

[0307] In one embodiment, when the processor 110 performs the step of determining the target offset based on the offset mapping relationship and the second grayscale image, it specifically executes the following steps:

[0308] Obtain the target environmental condition parameters when generating the second grayscale image, and obtain at least one reference calibration region of the reference calibration image;

[0309] Based on the target environmental condition parameters, the target region offset corresponding to each of the reference calibration regions is determined in the offset mapping relationship;

[0310] The pixel depth correction processing of the second grayscale image based on the target offset and the calibration reference image includes:

[0311] The target region offset is used to perform offset correction processing on the reference calibration region of the calibration reference image to obtain the offset-corrected calibration reference image. Pixel depth calculation processing is then performed on the second grayscale image based on the calibration reference image; or,

[0312] The target grayscale region corresponding to the reference calibration region in the second grayscale image is determined, and the target grayscale region of the second grayscale image is offset and corrected using the target region offset to obtain the second grayscale image after offset correction. Pixel depth calculation is performed on the second grayscale image based on the calibration reference image.

[0313] In one embodiment, the environmental condition parameters include at least one of temperature parameters, humidity parameters, scene parameters, and time parameters.

[0314] In one or more embodiments of this specification, the target offset for the calibration reference image can be determined based on the offset mapping relationship, and then the structured light module can be subjected to depth correction processing. This can significantly reduce or even eliminate the impact of uncertainties such as vibration, deformation, and temperature on the depth calculation scene. Furthermore, in scenarios where the calibration reference image is difficult to modify, the target offset of the calibration reference image determined based on the offset mapping relationship can overcome large offsets in the calibration reference image, improving the offset robustness in the depth calculation scene and ensuring the quality of depth calculation. Moreover, even when uncertainties cause large offsets to the calibration reference image and the user lacks the ability to update the calibration reference image, the depth calculation accuracy of the structured light module can still be guaranteed, ensuring its shooting performance. Furthermore, by incorporating environmental condition parameters such as temperature, humidity, scene, and time during the construction of the offset mapping relationship, the impact of these uncertainties on the depth calculation scene can be offset or mitigated, further improving the effect of depth offset correction and achieving offset quantification of the influence of uncertain environments.

[0315] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory, or random access memory, etc.

[0316] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. An image processing method, the method comprising: Acquire a first grayscale image captured by the structured light module, and acquire a calibration reference image corresponding to the structured light module, wherein the calibration reference image is a grayscale image generated by the structured light module at a calibration distance from the structured light module; Perform at least one round of image offset processing on the calibration reference image to obtain the offset calibration image after each round of image offset processing; Based on the offset calibration image and the first grayscale image in each round, an offset mapping relationship is determined for the calibration reference image; In the depth calculation scenario, the target offset for the calibration reference image is determined based on the offset mapping relationship, and the structured light module is subjected to depth correction processing based on the target offset; The step of determining the offset mapping relationship for the calibration reference image based on the offset calibration image and the first grayscale image in each round includes: Determine the image correlation between the offset calibration image and the first grayscale image in each round; Determine the target correlation from each of the image correlations, and obtain the image offset relative to the calibration reference image indicated by the target correlation; The offset mapping relationship for the calibration reference image is determined based on the image offset.

2. The method according to claim 1, wherein performing at least one round of image offset processing on the calibration reference image to obtain an offset calibration image after each round of image offset processing includes: Determine at least one reference calibration region for the calibration reference image in each round, and the reference offset corresponding to the reference calibration region; Based on the reference offset, each of the reference calibration regions of the calibration reference image is image offset to obtain at least one offset image region after each round of image offset processing and an offset calibration image containing the at least one offset image region.

3. The method according to claim 2, wherein determining at least one reference calibration region for each round of calibration reference image comprises: Obtain the first color image corresponding to the first grayscale image, wherein the image acquisition objects of the first grayscale image and the first color image are the same object; Perform image object region detection on the first color image to obtain at least one reference color region, and obtain the reference grayscale region of the reference color region in the first grayscale image. In the calibration reference image, a reference calibration region is determined for each round of the reference grayscale region.

4. The method according to claim 2, wherein the image offset of each of the reference calibration regions of the calibration reference image based on the reference offset comprises: Determine the offset adjustment value and offset adjustment direction corresponding to the reference offset, and perform image offset on each of the reference calibration regions along the offset adjustment direction using the offset adjustment value.

5. The method according to claim 4, wherein determining the image correlation between the offset calibration image and the first grayscale image in each round comprises: Obtain at least one offset image region corresponding to the offset calibration image in each round, and determine that the offset image region is in the reference grayscale region corresponding to the first grayscale image; Calculate the regional image correlation between the offset image region and the reference grayscale region, and use the regional image correlation in each round as the image correlation.

6. The method according to claim 4, wherein determining the target relevance from each of the image relevances comprises: The maximum correlation score is obtained from all the image correlation scores, and the maximum correlation score is used as the target correlation score.

7. The method according to claim 4, wherein determining the offset mapping relationship for the calibration reference image based on the image offset comprises: Construct a first offset mapping relationship between the image offset and the reference calibration area; And / or, Obtain the reference environmental condition parameters corresponding to the first grayscale image, and construct a second offset mapping relationship between the reference environmental condition parameters, the image offset, and the reference calibration area.

8. The method according to claim 7, when the number of first grayscale images is multiple, the step of obtaining reference environmental condition parameters corresponding to the first grayscale images and constructing a second offset mapping relationship between the reference environmental condition parameters, the image offset, and the reference calibration area includes: Determine the reference calibration area corresponding to the image offset, and obtain the reference environmental condition parameters corresponding to the first grayscale image; The at least one reference calibration region, the image offset of the reference calibration region, and the reference environmental condition parameters are used as offset mapping data for each of the first grayscale images; Based on the offset mapping data, a second offset mapping relationship is constructed between the reference environmental condition parameters, the image offset, and the reference calibration area.

9. The method according to claim 8, wherein constructing a second offset mapping relationship between the reference environmental condition parameters, the image offset, and the reference calibration region based on each of the offset mapping data comprises: In each of the offset mapping data, at least one first image offset corresponding to the same reference calibration region is obtained; The first image offsets are fused according to the reference environmental condition parameters to obtain the second image offset for the reference calibration area. Construct a second offset mapping relationship corresponding to the second image offset, the reference calibration region, and the reference environmental condition parameters.

10. The method according to claim 1, further comprising, after acquiring the first grayscale image of the structured light module and the calibration reference image corresponding to the structured light module: The first grayscale image is subjected to image desensitization processing to obtain the first grayscale image after image desensitization processing; And / or, The calibration reference image is subjected to image desensitization processing to obtain the calibration reference image after image desensitization processing.

11. The method according to claim 1, wherein determining the target offset for the calibration reference image based on the offset mapping relationship, and performing depth correction processing on the structured light module based on the target offset, comprises: The structured light module is used to obtain the second grayscale image of the target object. Based on the offset mapping relationship and the second grayscale image, the target offset for the calibration reference image is determined; The second grayscale image is subjected to pixel depth correction processing based on the target offset and the calibration reference image.

12. The method according to claim 11, wherein the offset mapping relationship includes a second offset mapping relationship based on reference environmental condition parameters, image offset, and reference calibration area, and the step of determining the target offset based on the offset mapping relationship and the second grayscale image includes: Obtain the target environmental condition parameters when generating the second grayscale image, and obtain at least one reference calibration region of the reference calibration image; Based on the target environmental condition parameters in the second offset mapping relationship, the target area offset corresponding to each of the reference calibration areas is determined; The pixel depth correction processing of the second grayscale image based on the target offset and the calibration reference image includes: The target region offset is used to perform offset correction processing on the reference calibration region of the calibration reference image to obtain the offset-corrected calibration reference image, and the pixel depth calculation processing is performed on the second grayscale image based on the calibration reference image; or, The target grayscale region corresponding to the reference calibration region in the second grayscale image is determined, and the target grayscale region of the second grayscale image is offset and corrected using the target region offset to obtain the second grayscale image after offset correction. Pixel depth calculation is performed on the second grayscale image based on the calibration reference image.

13. An image processing apparatus, the apparatus comprising: The image acquisition module is used to acquire a first grayscale image collected by the structured light module, and to acquire a calibration reference image corresponding to the structured light module. The calibration reference image is a grayscale image generated by the structured light module at a calibration distance from the structured light module. The relationship determination module is used to perform at least one round of image offset processing on the calibration reference image to obtain an offset calibration image after each round of image offset processing, and to determine the offset mapping relationship for the calibration reference image based on the offset calibration image and the first grayscale image in each round. The depth calculation module is used to determine the target offset for the calibration reference image based on the offset mapping relationship in the depth calculation scene, and to perform depth correction processing on the structured light module based on the target offset. The step of determining the offset mapping relationship for the calibration reference image based on the offset calibration image and the first grayscale image in each round includes: Determine the image correlation between the offset calibration image and the first grayscale image in each round; Determine the target correlation from each of the image correlations, and obtain the image offset relative to the calibration reference image indicated by the target correlation; The offset mapping relationship for the calibration reference image is determined based on the image offset.

14. A computer storage medium storing a plurality of instructions adapted for loading by a processor and executing the method steps of any one of claims 1 to 12.

15. A computer program product storing at least one instruction, said at least one instruction being loaded by a processor and executing the method steps of any one of claims 1 to 12.

16. An electronic device comprising: A processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and executed the method steps as claimed in any one of claims 1 to 12.

Images