Image color restoration method and device based on fusion of multi-camera system and light sensor array

Abstract

The invention provides an image color restoration method and device based on fusion of a multi-camera system and a light sensor array. The method comprises the following steps: firstly, on the basis of a binocular vision principle, performing matching arrangement on view field ranges of a light sensor and double cameras to generate a camera sensor integrated unit; afterwards, based on the camera sensor integration unit, calibration parameters are generated by calibrating the light sensor. And carrying out calibration mapping on the light sensor array and the camera to generate a mapping database. And setting the imaging parameters of the camera through the optical parameters captured by the light sensor, and generating a collected image. And finally, based on the mapping database, color information processing is carried out on the acquired image through an image processor, and color restoration of the image is completed. According to the embodiment of the invention, the fusion design of the light sensor array and the multi-camera system is realized, so that the precise detection of optical parameters is ensured, and the color consistency of the image is improved.

Description

Technical Field

[0001] The present application relates to the field of image technology, and in particular to an image color restoration method and device based on the fusion of a multi-camera system and a light sensor array. Background Art

[0002] In modern society, the demand for cameras in various fields is increasing, especially in the application of multi-camera systems, where the demand is exploding. These applications include but are not limited to multi-camera image fusion and stitching, multi-camera depth estimation based on binocular stereo principle, etc. With the widespread application of these technologies, image quality has gradually become the most important criterion for evaluating the performance of multi-camera systems, and color accuracy accounts for a particularly prominent proportion of image quality. Therefore, color accuracy and consistency have become key directions for research and development in the camera field.

[0003] There are two main existing color calibration methods: the first is to calibrate the color of each camera separately and adjust the color consistency of each camera through the compensation coefficient. Although this method is widely used and is the most basic and common practice, its scope of application is limited, especially in scenes with high color requirements or complex angles, the effect is not ideal. In addition, when the hardware differences of multi-camera systems are large, simple compensation coefficient adjustment cannot effectively solve the problem of color consistency. The second method is to perform image processing based on various algorithms and adjust the color with the help of mathematical models and calculation methods. However, these algorithms usually lack accurate physical measured data as a reference, resulting in limited applicability and unstable performance in complex environments. These methods often rely on theoretical models rather than real-world data, and it is difficult to achieve accurate color restoration.

[0004] Therefore, one or more methods are needed to solve the above problems.

[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0006] The present application shows a method and device for image color restoration based on the fusion of a multi-camera system and a light sensor array.

[0007] In a first aspect, the present application shows an image color restoration method based on the fusion of a multi-camera system and a light sensor array, the method comprising:

[0008] Based on the principle of binocular camera stereo vision, a camera sensor integrated unit is generated by matching the light sensor with the field of view of the dual cameras.

[0009] Based on the camera sensor integrated unit, generating calibration parameters by calibrating the light sensor;

[0010] Based on the calibration parameters, a mapping database is generated by calibrating and mapping the light sensor array and the camera;

[0011] Based on the stereo vision principle of binocular cameras, the camera imaging parameters are set through the optical parameters captured by the light sensor to generate a captured image;

[0012] Based on the mapping database, the image processor processes the color information of the captured image to complete the color restoration of the image.

[0013] In an exemplary embodiment of the present disclosure, the light sensor is arranged to match the field of view of the dual cameras, including:

[0014] The light sensor is provided with a lens, a lens barrel, a light averaging mirror, a light sensor chip, a circuit board, a power supply module, a data transmission module, and a signal transmission module;

[0015] Based on the overlapping range of the fields of view of the dual cameras, an array of light sensors is generated by arranging the light sensors at preset angles;

[0016] Based on the stereoscopic vision principle of binocular cameras, a camera sensor integrated unit is generated by arranging the light sensor array at the center of the baseline between the left camera and the right camera.

[0017] In an exemplary embodiment of the present disclosure, calibrating the light sensor includes:

[0018] Based on standard test conditions, generating test condition parameters by calibrating the consistency of optical parameters of the light sensor;

[0019] Based on the parameter compensation rule, generating compensation parameters by performing consistent parameter compensation on the test condition parameters;

[0020] Based on the compensation parameters, calibration parameters are generated by performing variable calibration on a single condition in the standard test conditions.

[0021] In an exemplary embodiment of the present disclosure, a light sensor array is calibrated and mapped with a camera, including:

[0022] Based on the principle of spatial alignment, a capture scene is generated by aligning the detection area of ​​the light sensor with the field of view of the camera;

[0023] Based on the standard test conditions, synchronously detecting the captured scene by the light sensor and the camera to generate calibration optical parameters, and mapping the calibration optical parameters to generate a calibration map;

[0024] Based on the calibration mapping, a mapping database is generated by performing calibration mapping of variables on a single condition in the standard test conditions.

[0025] In an exemplary embodiment of the present disclosure, the camera imaging parameters are set by using the optical parameters captured by the light sensor, including:

[0026] Based on the light sensor, an optical parameter is captured to generate an optical signal, and the optical signal is converted to generate a sensor digital signal;

[0027] Based on the digital signal processing method, the sensor digital signal is processed by a processor to generate light source frequency information.

[0028] In an exemplary embodiment of the present disclosure, the camera imaging parameters are set by using the optical parameters captured by the light sensor, including:

[0029] Based on the light source frequency information, an imaging parameter control signal is generated by controlling and modifying the camera imaging parameters;

[0030] Based on the imaging parameter control signal, the camera is controlled to collect images to generate a collected image.

[0031] In an exemplary embodiment of the present disclosure, color information processing is performed on the collected image, including:

[0032] Based on the stereo vision principle of binocular cameras, the image processor calculates the overlap area of ​​the dual-camera field of view range of the collected images to generate an overlap area image;

[0033] Based on the imaging quality requirement, the overlapping area image is cropped to generate a display area image;

[0034] Based on the mapping database, the color restoration of the image is completed by performing image color correction on the display area image.

[0035] In a second aspect, the present application shows an image color restoration device based on the fusion of a multi-camera system and a light sensor array, the device comprising:

[0036] Integrated unit design module, used to match the light sensor with the field of view of the dual cameras;

[0037] A sensor calibration module, used to calibrate the light sensor;

[0038] A database component module is used to calibrate and map the light sensor array and the camera;

[0039] An image acquisition module, used to set camera imaging parameters through optical parameters captured by the light sensor;

[0040] The image processing module is used to process the color information of the collected image through an image processor.

[0041] In a third aspect, the present application shows an electronic device, which includes: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute the method described in any of the above aspects.

[0042] In a fourth aspect, the present application illustrates a non-temporary computer-readable storage medium, which, when instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to execute a method as described in any of the above aspects.

[0043] In a fifth aspect, the present application illustrates a computer program product. When instructions in the computer program product are executed by a processor of an electronic device, the electronic device is enabled to execute the method described in any of the above aspects.

[0044] The technical solution provided by this application may have the following beneficial effects:

[0045] Based on the embodiment of the present disclosure, first, according to the principle of binocular vision, a camera sensor integrated unit is generated by matching and arranging the light sensor with the field of view of the dual cameras. Afterwards, based on the camera sensor integrated unit, calibration parameters are generated by calibrating the light sensor. Then, a mapping database is generated by calibrating and mapping the light sensor array and the camera. And, the camera imaging parameters are set by the optical parameters captured by the light sensor to generate a captured image. Finally, based on the mapping database, the image processor processes the color information of the captured image to complete the color restoration of the image. Therefore, the embodiment of the present disclosure provides a fusion design scheme of a light sensor array and a multi-camera system, which can restore the image color to the maximum extent, ensure the accurate detection of optical parameters, and improve the color consistency of the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 This is a flowchart of the steps of an image color restoration method based on the fusion of a multi-camera system and a light sensor array in the present application.

[0047] Figure 2This is a schematic diagram of the composition of a photosensitive sensor of an image color restoration method based on the fusion of a multi-camera system and a photosensitive sensor array in the present application.

[0048] Figure 3 This is a schematic diagram of the arrangement of a photosensitive sensor array according to an image color restoration method based on the fusion of a multi-camera system and a photosensitive sensor array in the present application.

[0049] Figure 4 This is a schematic diagram of the arrangement of cameras and light sensors in an image color restoration method based on the fusion of a multi-camera system and a light sensor array in the present application.

[0050] Figure 5 This is a schematic diagram of calculating the field of view range of cameras and light sensors in an image color restoration method based on the fusion of a multi-camera system and a light sensor array in the present application.

[0051] Figure 6 It is a schematic diagram of the final display area of ​​the camera and the light sensor of the image color restoration method based on the fusion of a multi-camera system and a light sensor array of the present application.

[0052] Figure 7 This is a structural block diagram of an image color restoration device based on the fusion of a multi-camera system and a light sensor array in the present application.

[0053] Figure 8 It is a block diagram of an electronic device of the present application.

[0054] Fig. 9 It is a block diagram of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0055] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0056] Reference Figure 1 , shows a flowchart of the steps of an image color restoration method based on the fusion of a multi-camera system and a light sensor array of the present application, the method can be applied to electronic devices, wherein the method specifically may include the following steps:

[0057] Step S101, based on the principle of binocular camera stereo vision, a camera sensor integrated unit is generated by matching and arranging the light sensor with the field of view of the dual cameras;

[0058] Step S102: Based on the camera sensor integrated unit, calibrate the light sensor to generate calibration parameters.

[0059] Step S103: Based on the calibration parameters, a mapping database is generated by calibrating and mapping the light sensor array and the camera.

[0060] Step S104, based on the stereoscopic vision principle of the binocular camera, the camera imaging parameters are set through the optical parameters captured by the light sensor to generate a captured image.

[0061] Step S105 , based on the mapping database, color information of the captured image is processed by an image processor to complete color restoration of the image.

[0062] A method for restoring image color based on the fusion of a multi-camera system and a light sensor array in an embodiment of the present disclosure will be further described.

[0063] In step S101, based on the principle of binocular camera stereo vision, a camera sensor integrated unit may be generated by matching and arranging a light sensor with the field of view of the dual cameras.

[0064] In some optional embodiments of this example, the composition of the light sensor is as follows: Figure 2 As shown, it includes: a lens, a lens barrel, a homogenizer, a light sensor chip, a circuit board, and other conventional modules such as a power supply module, a data transmission module, and a signal transmission module are not shown in the figure.

[0065] Specifically, the working principle of the light sensor is that the lens converges the light within the field of view angle θ, and after passing through the homogenizer, the light becomes a light spot with uniform light intensity distribution, which is received by the light sensor chip (the internal hardware and algorithm working method of the light sensor chip are not restricted, and the field of view angle θ of the light sensor module satisfies 0°<θ<0.5*camera field of view (FOV)). The light signal is converted into an electrical signal, and after amplification and filtering, the analog signal is converted into a sensor digital signal, which is finally transmitted to the processor. The processor can obtain the optical parameter information such as the modulation frequency, color temperature, and color coordinates of the light in the captured viewing angle through calculation.

[0066] After modulating the light sensor, Figure 3 As shown, a number n of light sensors are combined into a light sensor array.

[0067] Where n≥2, and in order to satisfy the requirement that the detection range of the light sensor matches the overlap range of the fields of view of the two cameras (i.e., the light sensor divides the overlap area of ​​the fields of view of the two cameras into n*n small areas), each light sensor in the light sensor array needs to be arranged at a certain angle. In a specific embodiment, the optical axis of each light sensor has angles α and β in both directions (the size of the angle is determined by the number of light sensors, the field of view angle, etc., but must satisfy 5°≤α<90°, 5°≤β<90°, and the photosensitive centers are in the same plane).

[0068] Afterwards, if Figure 4 As shown, m cameras (m≥3) are arranged with the optical axes in the same straight line. A group of light sensor arrays are arranged between every two cameras, and the light sensor arrays are located at the center of the baseline of the two cameras. In this way, the two cameras and the group of light sensors together constitute the core component of the embodiment of the present invention: the camera sensor integrated unit. The entire multi-camera system is composed of multiple groups of the camera sensor integrated units.

[0069] Based on the above method, by obtaining the field of view image of the overlapping area, the accuracy of measuring the object can be greatly improved. Combined with the arrangement of the light sensor array, the spot area captured by each light sensor is smaller, which further enhances the measurement accuracy, especially in color reproduction.

[0070] In step S102, calibration parameters may be generated by calibrating the light sensor based on the camera sensor integrated unit.

[0071] In some optional embodiments of this example, the parameters of the light sensor need to be calibrated before formal use.

[0072] Specifically, first, under standard test conditions such as the same light source, the same color temperature, the same illumination, the same detection distance, the same calibration color card (or other calibration objects), and the same angle as the calibration color card, the optical parameters such as light source frequency, color temperature, color coordinates, and brightness are uniformly calibrated for consistency, and the test condition parameters are output.

[0073] However, during the calibration process, not the test condition parameters of each light sensor are consistent. At this time, it is necessary to introduce parameter compensation rules. By performing parameter compensation on the test condition parameters output by each light sensor in the light sensor array, the compensation parameters generated by each light sensor in the light sensor matrix are consistent after parameter compensation.

[0074] Finally, according to the compensation parameters, the light source, color temperature, illumination, detection distance, calibration color card (or other calibration objects), and the angle with the calibration color card are changed separately and one-time, and then the optical parameters are calibrated for consistency through the changed variables to generate calibration parameters.

[0075] In this way, through multiple calibrations, it can be ensured that the calibration parameters output by each light sensor in the light sensor array are consistent with the actual light source parameters.

[0076] In step S103, a mapping database may be generated by calibrating and mapping the light sensor array and the camera based on the calibration parameters.

[0077] In some optional embodiments of this example, after the light sensor array is calibrated, the light sensor array and the camera need to be calibrated.

[0078] During the calibration process, the area detected by the light sensor is aligned with the field of view of the camera based on the principle of spatial alignment.

[0079] Afterwards, under the same standard test conditions (same light source, color temperature, illumination, detection distance, calibration card (or other calibration object)), the light sensor array and camera are used to detect the captured scene at the same time. The image data detected by the light sensor array and captured by the camera are then mapped to parameters such as color temperature, color coordinates and brightness.

[0080] Then, according to the calibration mapping relationship, a single condition variable in the standard test condition is changed (for example, the light source is changed alone), and the image data detected by the light sensor array and taken by the camera are newly collected to map parameters such as color temperature, color coordinates and brightness. A mapping database is established and stored in the memory.

[0081] In this way, the mapping database stores the correspondence between each light sensor detection area and the camera field of view area. The processor can read the mapping database to ensure that the image captured by the camera and the output data of the light sensor are mapped to each other, thereby providing accurate image information. This ensures that in subsequent operations, it can help to accurately calibrate the color and brightness of the image.

[0082] In step S104, based on the stereoscopic vision principle of a binocular camera, the camera imaging parameters may be set by the optical parameters captured by the light sensor to generate a captured image.

[0083] In some optional embodiments of this example, after completing the above-mentioned preliminary preparations, the light sensor and the camera can be started synchronously. Among them, the light sensor is responsible for capturing the optical parameters such as the modulation frequency and color temperature of the ambient light source. The camera that starts taking images synchronously can also ensure that there is no time deviation between the data collected by the light sensor and the image taken by the camera, thereby ensuring the timeliness and consistency of the image data.

[0084] Afterwards, the light sensor converts the captured light signal containing optical parameters into an electrical signal, and then undergoes amplification, filtering and other processing before converting it into a sensor digital signal and transmitting it to the processor.

[0085] The processor uses the received digital signal from the sensor to perform calculations to obtain light source modulation frequency, surface color temperature of the photographed object, color coordinates, illumination and other light source frequency information.

[0086] In some optional embodiments of this example, the processor determines the camera exposure time, frame rate and other imaging parameters according to the light source frequency information obtained by the above calculation, and generates an imaging parameter control signal when it is necessary to change the exposure time, frame rate and other imaging parameters to ensure that the color and brightness of the image captured by the camera are consistent with the ambient light source detected by the light sensor, and issues a new instruction to the camera based on the signal. The camera uses the new instruction to capture the current field of view area and obtain the captured image.

[0087] When there is no need to change imaging parameters such as exposure time and frame rate, no new instructions are issued. The camera uses the previous instructions to shoot the current field of view area and obtain the collected image.

[0088] In step S105, the color information of the captured image may be processed by an image processor based on the mapping database to complete color restoration of the image.

[0089] In some optional embodiments of this example, Figure 5 As shown, after the images are captured by each camera, they are transmitted to the image processor for color restoration.

[0090] In a specific example, the image is processed by the ISP image processor.

[0091] In the first step, based on the collected image, the distance Z of the photographed object can be measured by the stereo vision principle of the binocular camera.

[0092] The second step is to calculate the length of the long side of the left camera horizontally by aL=2*Z*Tan(ωL / 2);

[0093] The third step is to calculate the length of the long side of the horizontal aspect of the right camera by aR=2*Z*Tan(ωR / 2);

[0094] Step 4: Calculate the length of the long side of the horizontal aspect of the overlapped area image by dH=(aL / 2+b+aR / 2)-2*((aL / 2+b / 2)-(aR / 2-b / 2));

[0095] Step 5: Calculate the length of the short side of the left camera in the vertical direction by cL=2*Z*Tan(ψL / 2);

[0096] Step 6: Calculate the length of the short side of the right camera in the vertical direction by cR=2*Z*Tan(ψR / 2);

[0097] Step 7: Calculate the length of the short side of the vertical aspect of the overlapped area image by dV=(cL / 2+b+cR / 2)-2*((cL / 2+b / 2)-(cR / 2-b / 2)).

[0098] Among them, ωL is the horizontal field of view of the left camera (known parameter), ωR is the horizontal field of view of the right camera (known parameter), ψL is the vertical field of view of the left camera (known parameter), ψR is the vertical field of view of the right camera (known parameter), and b is the distance between the optical axes of the two cameras, that is, the baseline length (known parameter).

[0099] Of course, in actual operation, there are also cases where the field of view angles overlap in the vertical direction (horizontal direction), that is, when ψL=ψR(ωL=ωR). In this case, the calculated vertical (horizontal) side length of the left camera or the vertical (horizontal) side length of the right camera can be directly substituted into the calculation formula for the side length of the overlapped area image to obtain the vertical (horizontal) side length of the overlapped area image.

[0100] After obtaining the overlapping area image, based on the imaging quality requirements, such as Figure 6 As shown, the overlapped area image is cut again to obtain a final display area image, wherein the length X of the display area image is less than dH, and the width Y is less than dV.

[0101] After obtaining the display area image, the calibration data stored in the previous mapping database is called to calculate the image color. At the same time, both cameras use the light sensor array as a reference to ensure that the image color in this area is consistent.

[0102] Finally, the entire multi-camera system uses the above-mentioned camera sensor integrated unit as a basic unit to perform color correction on the images collected by the entire multi-camera system to complete the color restoration of the images.

[0103] In this way, the light sensor module and array designed in this solution achieve color consistency between multi-camera systems. At the same time, the customized light sensor array detects information such as light source frequency and color temperature, accurately matches the camera imaging area, eliminates flicker, and improves color reproduction. In addition, the system optimizes the imaging area through binocular camera depth estimation, and combines hardware calibration to ensure the consistency of image color of each basic unit. This method can effectively improve the color accuracy of multi-camera systems, ensure consistent image quality, and is suitable for a variety of shooting needs.

[0104] It should be noted that, for the method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions involved are not necessarily required by the present application.

[0105] Reference Figure 7 , shows a structural block diagram of an image color restoration device 500 based on the fusion of a multi-camera system and a light sensor array of the present application, the device comprising:

[0106] An integrated unit design module 501 is used to match and arrange the light sensor with the field of view of the dual cameras;

[0107] A sensor calibration module 502, used to calibrate the light sensor;

[0108] A database component module 503 is used to calibrate and map the light sensor array and the camera;

[0109] An image acquisition module 504, used to set camera imaging parameters through optical parameters captured by the light sensor;

[0110] The image processing module 505 is used to process the color information of the collected image through an image processor.

[0111] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0112] Optionally, an embodiment of the present application further provides an electronic device, comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the various processes of the above-mentioned method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0113] The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, each process of the above method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0114] Figure 8 800 is a block diagram of an electronic device 800 shown in the present application. For example, the electronic device 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0115] Reference Figure 8 , the electronic device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0116] The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0117] The memory 804 is configured to store various types of data to support operations on the device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phone book data, messages, images, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0118] The power supply component 806 provides power to the various components of the electronic device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 800.

[0119] The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and the rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0120] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the electronic device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0121] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0122] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the electronic device 800. For example, the sensor assembly 814 can detect the open / closed state of the device 800, the relative positioning of components, such as the display and keypad of the electronic device 800, and the sensor assembly 814 can also detect the position change of the electronic device 800 or a component of the electronic device 800, the presence or absence of contact between the user and the electronic device 800, the orientation or acceleration / deceleration of the electronic device 800, and the temperature change of the electronic device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0123] The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as WiFi, a carrier network (such as 2G, 3G, 4G or 5G), or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast operation information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0124] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0125] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, and the instructions can be executed by a processor 820 of an electronic device 800 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0126] Fig. 919 is a block diagram of a computer-readable storage medium 1900 shown in the present application. For example, the computer-readable storage medium 1900 may be provided as a server.

[0127] Reference Fig. 9 , the computer-readable storage medium 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions, such as an application, that can be executed by the processing component 1922. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0128] The computer readable storage medium 1900 may also include a power supply component 1926 configured to perform power management of the computer readable storage medium 1900, a wired or wireless network interface 1950 configured to connect the computer readable storage medium 1900 to a network, and an input / output (I / O) interface 1958. The computer readable storage medium 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™ or the like.

[0129] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0130] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0131] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

[0132] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the present application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0133] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0134] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0135] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0136] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0137] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks, or optical disks.

[0138] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An image color restoration method based on the fusion of a multi-camera system and a light sensor array, characterized in that: The method comprises: Based on the principle of binocular camera stereo vision, a camera sensor integrated unit is generated by matching the light sensor with the field of view of the dual cameras. Based on the camera sensor integrated unit, generating calibration parameters by calibrating the light sensor; Based on the calibration parameters, a mapping database is generated by calibrating and mapping the light sensor array and the camera; Based on the stereo vision principle of binocular cameras, the camera imaging parameters are set through the optical parameters captured by the light sensor to generate a captured image; Based on the mapping database, the image processor processes the color information of the captured image to complete the color restoration of the image.

2. The method according to claim 1, characterized in that By matching the light sensor with the field of view of the dual cameras, the following are included: The light sensor is provided with a lens, a lens barrel, a light averaging mirror, a light sensor chip, a circuit board, a power supply module, a data transmission module, and a signal transmission module; Based on the overlapping range of the fields of view of the dual cameras, an array of light sensors is generated by arranging the light sensors at preset angles; Based on the stereoscopic vision principle of binocular cameras, a camera sensor integrated unit is generated by arranging the light sensor array at the center of the baseline between the left camera and the right camera.

3. The method according to claim 1, characterized in that The light sensor is calibrated, including: Based on standard test conditions, generating test condition parameters by calibrating the consistency of optical parameters of the light sensor; Based on the parameter compensation rule, generating compensation parameters by performing consistent parameter compensation on the test condition parameters; Based on the compensation parameters, calibration parameters are generated by performing variable calibration on a single condition in the standard test conditions.

4. The method according to claim 3, characterized in that By calibrating and mapping the light sensor array with the camera, including: Based on the principle of spatial alignment, a capture scene is generated by aligning the detection area of ​​the light sensor with the field of view of the camera; Based on the standard test conditions, synchronously detecting the captured scene by the light sensor and the camera to generate calibration optical parameters, and mapping the calibration optical parameters to generate a calibration map; Based on the calibration mapping, a mapping database is generated by performing calibration mapping of variables on a single condition in the standard test conditions.

5. The method according to claim 1, characterized in that The camera imaging parameters are set by using the optical parameters captured by the light sensor, including: Based on the light sensor, an optical parameter is captured to generate an optical signal, and the optical signal is converted to generate a sensor digital signal; Based on the digital signal processing method, the sensor digital signal is processed by a processor to generate light source frequency information.

6. The method according to claim 5, characterized in that The camera imaging parameters are set by using the optical parameters captured by the light sensor, including: Based on the light source frequency information, an imaging parameter control signal is generated by controlling and modifying the camera imaging parameters; Based on the imaging parameter control signal, the camera is controlled to collect images to generate a collected image.

7. The method according to claim 1, characterized in that By performing color information processing on the collected image, including: Based on the stereo vision principle of binocular cameras, the image processor calculates the overlap area of ​​the dual-camera field of view range of the collected images to generate an overlap area image; Based on the imaging quality requirement, the overlapping area image is cropped to generate a display area image; Based on the mapping database, the color restoration of the image is completed by performing image color correction on the display area image.

8. An image color restoration device based on the fusion of a multi-camera system and a light sensor array, characterized in that: include: Integrated unit design module, used to match the light sensor with the field of view of the dual cameras; A sensor calibration module, used to calibrate the light sensor; A database component module is used to calibrate and map the light sensor array and the camera; An image acquisition module, used to set camera imaging parameters through optical parameters captured by the light sensor; The image processing module is used to process the color information of the collected image through an image processor.

9. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the method according to any one of claims 1 to 7 when executed by the processor.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images