Image signal processing method, image processing device, equipment and storage medium
By acquiring the exposure parameters and lighting status of the image sensor in real time, and dynamically controlling the on- or off of the image processing module, the real-time and energy consumption problems of image signal processing in the autonomous driving system are solved, and the stability and safety of the system are improved.
Patent Information
- Application Number
- CN202410959774.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-17
- Publication Date
- 2024-11-08
AI Technical Summary
The problem of poor real-time performance and high energy consumption of image signal processing in autonomous driving systems.
Image data is collected in real time through the image sensor, exposure parameters are obtained, light level values and states are determined, and the image processing module is turned on or off according to the light state to adapt to the current image processing needs, improve image quality and reduce power consumption.
It improves the real-time and stability of the autonomous driving system, reduces image transmission delay and power consumption, and ensures image processing quality and system security under different lighting conditions.
Smart Images

Figure CN118921559A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing technology, and in particular to an image signal processing method, an image processing device, equipment and a storage medium. Background Art
[0002] The Image Signal Processing (ISP) system is used to process digital image signals, mainly including image denoising, color interpolation, image enhancement and other modules, and plays a vital role in the autonomous driving system. The image signal processing system is responsible for processing image information from image sensors around the vehicle so that the autonomous driving system can perform real-time environmental perception and decision-making. Therefore, the quality of the image generated by the image signal processing system is crucial to the autonomous driving system.
[0003] At present, the ISP (image signal processing) of autonomous driving faces many challenges and difficulties, among which the most prominent ones are the challenges of real-time performance and energy saving.
[0004] Regarding the real-time issue, autonomous driving requires image processing to have extremely low latency, and image signal processing usually needs to be completed within milliseconds. Currently, the ISP latency of autonomous driving is high, and real-time performance cannot be guaranteed, that is, the image obtained after image signal processing cannot be delivered to the autonomous driving system in real time.
[0005] In terms of energy saving, since energy efficiency is crucial to the sustainability and feasibility of autonomous driving systems, the energy consumption of ISPs in existing autonomous driving systems is relatively high.
[0006] Therefore, it is necessary to provide an image signal processing method, an image processing device, an equipment and a storage medium to solve the problems of poor real-time performance and high energy consumption of image signal processing in the autonomous driving system in the above-mentioned prior art. Summary of the invention
[0007] The purpose of the present invention is to provide an image signal processing method, an image processing device, an equipment and a storage medium to solve the problems of poor real-time performance and high energy consumption of image signal processing in an autonomous driving system.
[0008] To achieve the above object, the image signal processing method provided by the present invention comprises:
[0009] Acquiring image data of the current time in real time through an image sensor to obtain real-time image data, and obtaining exposure parameters of the image sensor corresponding to the real-time image data;
[0010] Determine the light level value at the current time according to the exposure parameter;
[0011] Acquire the lighting state according to the lighting level value and a preset threshold;
[0012] Controlling the opening or closing of a plurality of image processing modules according to the illumination state to obtain a processed image;
[0013] The processed image is transmitted to the autonomous driving system.
[0014] Optionally, the step of determining the light level value at the current time according to the exposure parameter comprises:
[0015] Obtain the exposure time and exposure gain in the exposure parameters,
[0016] A multiplication operation is performed on the exposure time and the exposure gain to obtain the light level value.
[0017] Optionally, the preset threshold includes a first threshold and a second threshold, and the step of acquiring the illumination state according to the illumination level value and the preset threshold includes:
[0018] A comparison operation is performed on the light level value and the first threshold and the second threshold to obtain a light state, wherein the first threshold is greater than the second threshold.
[0019] Optionally, the step of performing a comparison operation on the light level value with the first threshold value and the second threshold value to obtain the light state includes:
[0020] When the preset threshold is greater than or equal to the first threshold, the lighting state is determined to be a poor lighting state.
[0021] Optionally, the step of controlling the opening or closing of a plurality of image processing modules according to the illumination state to obtain a processed image includes:
[0022] After determining the poor illumination state, a first control instruction is sent to the plurality of image processing modules to control the plurality of image processing modules to be in an on state.
[0023] Optionally, the step of performing a comparison operation on the light level value with the first threshold value and the second threshold value to obtain the light state includes:
[0024] When the preset threshold is greater than or equal to the second threshold and less than the first threshold, it is determined that the lighting state is a normal lighting state.
[0025] Optionally, the step of controlling the opening or closing of a plurality of image processing modules according to the illumination state to obtain a processed image includes:
[0026] After determining the general state of illumination, a second control instruction is sent to the plurality of image processing modules to control the image noise reduction unit and the color interpolation unit to be in an on state, and to control the image enhancement unit to be in a off state.
[0027] Optionally, the step of performing a comparison operation on the light level value with the first threshold value and the second threshold value to obtain the light state includes:
[0028] When the preset threshold is less than or equal to the second threshold, it is determined that the lighting state is a good lighting state.
[0029] Optionally, the step of controlling the opening or closing of a plurality of image processing modules according to the illumination state to obtain a processed image includes:
[0030] After determining that the illumination is in a good state, a third control instruction is sent to the plurality of image processing modules to control the color interpolation unit to be in an on state, and to control the image noise reduction unit and the image enhancement unit to be in an off state.
[0031] The present invention also provides an image processing device, comprising
[0032] A data acquisition module, configured to acquire exposure parameters of the image sensor corresponding to the real-time image after the image sensor acquires image data of the current time in real time to acquire the real-time image data;
[0033] A calculation module, used for determining the light level value at the current time according to the exposure parameter;
[0034] A state determination module, used for obtaining the light state according to the light level value and a preset threshold;
[0035] A control module, used for controlling the opening or closing of a plurality of image processing modules according to the illumination state to obtain a processed image;
[0036] A transmission module is used to transmit the processed image to the automatic driving system.
[0037] Optionally, the image processing module includes an image noise reduction unit, a color interpolation unit and an image enhancement unit.
[0038] The present invention also provides a device, comprising a memory and a processor, wherein a program is stored in the processor, and when the program is executed by the processor, the image signal processing method is implemented.
[0039] The present invention also provides a storage medium, wherein a program is stored in the storage medium, and when the program is executed by a processor, the image signal processing method is implemented.
[0040] The image signal processing method, image processing device, equipment and storage medium provided by the present invention have the following beneficial effects:
[0041] The image signal processing method of the present invention obtains the exposure parameters of the image sensor corresponding to the real-time image, determines the light level value at the current time according to the exposure parameters, and obtains the light state according to the light level value and a preset threshold value; controls the opening or closing of several image processing modules according to the light state to obtain the processed image; the image processing module can be selectively opened or closed according to the light state, so as to meet the image processing requirements of the current real-time image data, and the image processing module can be opened when the light state is not good to increase the image quality, thereby ensuring the stable and accurate operation of the automatic driving system;
[0042] The image signal processing method of the present invention can selectively shut down one or several modules among several image processing modules when the lighting conditions are good, thereby reducing the running time required for image processing, reducing image transmission delay, and improving the stability and safety of the operation of the automatic driving system. The real-time performance is good, and because some image processing modules are shut down when the current lighting conditions are good, the purpose of reducing image processing power consumption is achieved. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] Figure 1 A flowchart of an image signal processing method provided by an embodiment of the present invention;
[0044] Figure 2 FIG. 4 is a schematic diagram of the structure of an image processing device according to an embodiment of the present invention. DETAILED DESCRIPTION
[0045] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. Unless otherwise defined, the technical terms or scientific terms used herein should be understood by people with general skills in the field to which the present invention belongs. "Including" and similar words used in this article mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects.
[0046] In view of the problems existing in the prior art, the embodiments of the present invention provide an image signal processing method, an image processing apparatus, a device and a storage medium.
[0047] Reference Figure 1The image signal processing method provided by the present invention comprises:
[0048] S1: collecting image data of the current time in real time through an image sensor to obtain real-time image data, and obtaining exposure parameters of the image sensor corresponding to the real-time image data;
[0049] S2: Determine the light level value at the current time according to the exposure parameters;
[0050] S3: Obtaining the lighting state according to the lighting level value and the preset threshold;
[0051] S4: controlling the opening or closing of a plurality of image processing modules according to the illumination state to obtain a processed image;
[0052] S5: Transmit the processed image to the autonomous driving system.
[0053] As an optional implementation of the embodiment of the present invention, in step S2, the step of determining the light level value at the current time according to the exposure parameter includes:
[0054] Get the exposure time and exposure gain in the exposure parameters.
[0055] Multiply the exposure time and exposure gain to get the light level value.
[0056] In some embodiments, the light level value can be obtained by the following calculation formula:
[0057] LL=ET×EG
[0058] Among them, LL is the light level value, ET is the exposure time, and EG is the exposure gain.
[0059] As an optional implementation of the embodiment of the present invention, in step S3, the preset threshold includes a first threshold and a second threshold, and the step of acquiring the illumination state according to the illumination level value and the preset threshold includes:
[0060] S30: performing a comparison operation on the light level value and a first threshold and a second threshold to obtain a light state, wherein the first threshold is greater than the second threshold.
[0061] In a first implementation of the embodiment of the present invention, in step S30, the step of performing a comparison operation on the light level value and the first threshold and the second threshold to obtain the light state includes:
[0062] S31: When the preset threshold is greater than or equal to the first threshold, determining that the lighting state is a poor lighting state.
[0063] In some embodiments, in step S4, the step of controlling the opening or closing of a plurality of image processing modules according to the illumination state to obtain a processed image includes:
[0064] S41: After determining that the illumination is poor, sending a first control instruction to a plurality of image processing modules to control the plurality of image processing modules to be in an on state.
[0065] In some embodiments, several image processing modules include an image denoising unit, a color interpolation unit and an image enhancement unit. The image denoising unit is used to perform denoising on real-time images, the color interpolation unit is used to perform color interpolation on real-time image data, and the image enhancement unit is used to perform image enhancement on real-time images. Their specific structures and image processing methods are common knowledge among people in this field and will not be elaborated here.
[0066] In some specific embodiments, the exposure parameters of the image sensor detected and obtained are exposure time ET=3, exposure gain EG=8, and the light level value LL=24 is calculated by step S2; the first threshold thres1=15 and the second threshold thres2=5 are set;
[0067] If it is determined in step S3 that LL>thres1, it is determined that the lighting state is a poor lighting state. At this time, step S41 is entered to send a first control instruction to several image processing modules so that the image noise reduction unit, the color interpolation unit and the image enhancement unit are all turned on to ensure the quality of image signal processing.
[0068] It can be explained that the first threshold and the second threshold can be preset into the image processing module through programming to facilitate the determination of the current lighting state, and can also be set and adjusted according to the needs of image processing, which will not be repeated here.
[0069] In a second implementation of the embodiment of the present invention, in step S30, the step of performing a comparison operation on the light level value and the first threshold and the second threshold to obtain the light state includes:
[0070] S32: When the preset threshold is greater than or equal to the second threshold and less than the first threshold, determining that the illumination state is a normal illumination state.
[0071] In a second implementation of the embodiment of the present invention, in step S4, the step of controlling the opening or closing of a plurality of image processing modules according to the illumination state to obtain a processed image includes:
[0072] S42: After determining the general state of illumination, send a second control instruction to a plurality of image processing modules to control the image noise reduction unit and the color interpolation unit to be turned on, and control the image enhancement unit to be turned off.
[0073] In some specific embodiments, the exposure parameters of the image sensor detected and obtained are exposure time ET=1, exposure gain EG=8, and the light level value LL=8 is calculated in step S2; the first threshold thres1=15 and the second threshold thres2=5 are set;
[0074] By judging through step S3 that thres2<LL<thres1, it is determined that the lighting state is a poor lighting state. At this time, step S42 is entered to send a second control instruction to several image processing modules. Some image processing units in the several image processing modules can be selectively turned off, such as keeping the image noise reduction unit and the color interpolation unit in the turned-on state, and turning off the image enhancement unit, thereby achieving the purpose of reducing power consumption and reducing the delay of image signal processing, and improving the real-time performance of image signal processing.
[0075] It can be explained that the above-mentioned on state refers to the state in which the units in the image processing module are started and maintained in stable operation, and the off state refers to the state in which the units in the image processing module are shut down or in standby state, so as to ensure the quality of image signal processing while avoiding power consumption waste caused by too many units being turned on in image signal processing.
[0076] In a third implementation of the embodiment of the present invention, in step S30, the step of performing a comparison operation on the light level value and the first threshold and the second threshold to obtain the light state includes:
[0077] S33: When the preset threshold is less than or equal to the second threshold, determining that the lighting state is a good lighting state.
[0078] In some embodiments, in step S4, the step of controlling the opening or closing of a plurality of image processing modules according to the illumination state to obtain a processed image includes:
[0079] S43: After determining that the lighting is in a good state, a third control instruction is sent to a plurality of image processing modules to control the color interpolation unit to be in an on state, and to control the image noise reduction unit and the image enhancement unit to be in an off state.
[0080] In some specific embodiments, the exposure parameters of the image sensor detected and obtained are respectively exposure time ET=1, exposure gain EG=1, then the light level value LL=1 is calculated by step S2; the first threshold thres1=15 and the second threshold thres2=5 are set;
[0081] If it is determined in step S3 that LL<thres2, it is determined that the illumination state is a poor illumination state. At this time, step S43 is entered to send a third control instruction to several image processing modules, and some image processing units in the several image processing modules can be selectively turned off, such as keeping the color interpolation unit turned on, and turning off the image noise reduction unit and the image enhancement unit, thereby achieving the purpose of reducing power consumption and reducing the delay of image signal processing, and improving the real-time performance of image signal processing.
[0082] The present invention also provides an image processing device, referring to Figure 2 , the image processing device includes a data acquisition module 1, which is used to acquire the exposure parameters of the image sensor 9 corresponding to the real-time image after the image sensor 9 acquires the image data of the current time in real time to acquire the real-time image data;
[0083] A calculation module 2, used for determining the light level value at the current time according to the exposure parameters;
[0084] A state determination module 3, used for obtaining the light state according to the light level value and a preset threshold;
[0085] A control module 4, used to control the opening or closing of a plurality of image processing modules according to the illumination state to obtain a processed image;
[0086] The transmission module 5 is used to transmit the processed image to the automatic driving system.
[0087] In some embodiments, the data acquisition module 1 is used to detect and acquire exposure parameters of the image sensor 9 .
[0088] In some embodiments, the step of calculating module 2 determining the light level value at the current time according to the exposure parameter includes:
[0089] Get the exposure time and exposure gain in the exposure parameters.
[0090] Multiply the exposure time and exposure gain to get the light level value.
[0091] In some specific embodiments, the light level value can be obtained by the following calculation formula:
[0092] LL=ET×EG
[0093] Among them, LL is the light level value, ET is the exposure time, and EG is the exposure gain.
[0094] In some embodiments, the preset threshold includes a first threshold and a second threshold, and the step of the state determination module 3 acquiring the illumination state according to the illumination level value and the preset threshold includes:
[0095] A comparison operation is performed on the light level value and a first threshold and a second threshold to obtain the light state, wherein the first threshold is greater than the second threshold.
[0096] In some embodiments, the step of performing a comparison operation on the light level value with the first threshold and the second threshold to obtain the light state includes:
[0097] When the preset threshold is greater than or equal to the first threshold, determining that the lighting state is a poor lighting state;
[0098] When the preset threshold is greater than or equal to the second threshold and less than the first threshold, determining that the illumination state is a normal illumination state;
[0099] When the preset threshold is less than or equal to the second threshold, it is determined that the lighting state is a good lighting state.
[0100] In some embodiments, the control module 4 may be a single chip microcomputer.
[0101] In some embodiments, the transmission module 5 is a module for transmitting image signals, which can be transmitted to the automatic driving system using a data transmission line or a data output port of an image processing module. Since data transmission is a conventional technology for those skilled in the art, it will not be described in detail here.
[0102] In some embodiments, reference Figure 2 The control module 4 controls the opening or closing of several image processing modules according to the illumination state to obtain the processed image, and the steps include:
[0103] After the state confirmation module 3 determines that the illumination is poor, the control module 4 sends a first control instruction to the plurality of image processing modules to control the plurality of image processing modules to be in an on state, that is, the image noise reduction unit 6, the color interpolation unit 7 and the image enhancement unit 8 are all in an on state;
[0104] After the state confirmation module 3 determines the general state of the illumination, the control module 4 sends a second control instruction to the plurality of image processing modules to control the image noise reduction unit 6 and the color interpolation unit 7 to be turned on, and to control the image enhancement unit 8 to be turned off;
[0105] After the state confirmation module 3 determines that the illumination is good, the control module 4 sends a third control instruction to the image processing modules to control the color interpolation unit 7 to be turned on, and control the image noise reduction unit 6 and the image enhancement unit 8 to be turned off.
[0106] In some specific embodiments, the exposure parameters of the image sensor detected and acquired by the data acquisition module 1 are respectively exposure time ET=3, exposure gain EG=8, and the light level value LL=24 is calculated by the calculation module 2; the first threshold thres1=15 and the second threshold thres2=5 are set;
[0107] The state confirmation module 3 determines that LL>thres1, and determines that the illumination state is a poor illumination state. At this time, the control module 4 sends a first control instruction to several image processing modules, so that the image noise reduction unit, the color interpolation unit and the image enhancement unit are all in the turned-on state to ensure the quality of image signal processing;
[0108] In some other specific embodiments, the exposure parameters of the image sensor detected and acquired by the data acquisition module 1 are respectively exposure time ET=1, exposure gain EG=8, and the light level value LL=8 is calculated by the calculation module 2; the first threshold thres1=15 and the second threshold thres2=5 are set;
[0109] If the state confirmation module 3 determines that thres2<LL<thres1, it determines that the lighting state is a poor lighting state. At this time, the control module 4 sends a second control instruction to several image processing modules to control the image noise reduction unit and the color interpolation unit to remain in the turned-on state, and turn off the image enhancement unit, thereby achieving the purpose of reducing power consumption and reducing the delay of image signal processing, and improving the real-time performance of image signal processing.
[0110] In some further specific embodiments, the exposure parameters of the image sensor detected and obtained are respectively exposure time ET=1, exposure gain EG=1, and the light level value LL=1 is calculated by the calculation module 2; the first threshold thres1=15 and the second threshold thres2=5 are set;
[0111] If the state confirmation module 3 determines that LL<thres2, it determines that the lighting state is a poor lighting state. At this time, the control module 4 sends a third control instruction to several image processing modules to control the color interpolation unit to remain in the turned-on state, and turn off the image noise reduction unit and the image enhancement unit, thereby achieving the purpose of reducing power consumption and reducing the delay of image signal processing, and improving the real-time performance of image signal processing.
[0112] The present invention also provides a device, comprising a memory and a processor, wherein a program is stored in the processor, and when the program is executed by the processor, the image signal processing method is implemented.
[0113] The present invention also provides a storage medium, in which a program is stored, and when the program is executed by a processor, the image signal processing method is implemented.
[0114] In some embodiments, the storage medium may be any computer-readable storage medium such as a USB flash drive, a hard disk, or an optical disk, and is not limited here.
[0115] In summary, the advantages of the image signal processing method, image processing device, equipment and storage medium provided by the present invention are:
[0116] The image signal processing method of the present invention obtains the exposure parameters of the image sensor corresponding to the real-time image, determines the light level value at the current time according to the exposure parameters, and obtains the light state according to the light level value and a preset threshold value; controls the opening or closing of several image processing modules according to the light state to obtain the processed image; the image processing module can be selectively opened or closed according to the light state, so as to meet the image processing requirements of the current real-time image data, and the image processing module can be opened when the light state is not good to increase the image quality, thereby ensuring the stable and accurate operation of the automatic driving system;
[0117] The image signal processing method of the present invention can selectively shut down one or several modules among several image processing modules when the lighting conditions are good, thereby reducing the running time required for image processing, reducing image transmission delay, and improving the stability and safety of the operation of the automatic driving system. The real-time performance is good, and because some image processing modules are shut down when the current lighting conditions are good, the purpose of reducing image processing power consumption is achieved.
[0118] Although the embodiments of the present invention are described in detail above, it is obvious to those skilled in the art that various modifications and variations can be made to these embodiments. However, it should be understood that such modifications and variations are within the scope and spirit of the present invention as described in the claims. Moreover, the present invention described herein may have other embodiments and may be implemented or realized in a variety of ways.
Claims
1. A method for processing an image signal, It is characterized in that include: Acquiring image data of the current time in real time through an image sensor to obtain real-time image data, and obtaining exposure parameters of the image sensor corresponding to the real-time image data; Determine the light level value at the current time according to the exposure parameter; Acquire the lighting state according to the lighting level value and a preset threshold; Controlling the opening or closing of a plurality of image processing modules according to the illumination state to obtain a processed image; The processed image is transmitted to the autonomous driving system.
2. The image signal processing method according to claim 1, It is characterized in that The step of determining the light level value at the current time according to the exposure parameter comprises: Obtain the exposure time and exposure gain in the exposure parameters, A multiplication operation is performed on the exposure time and the exposure gain to obtain the light level value.
3. The image signal processing method according to claim 2, It is characterized in that The preset threshold value includes a first threshold value and a second threshold value, and the step of acquiring the illumination state according to the illumination level value and the preset threshold value includes: A comparison operation is performed on the light level value and the first threshold and the second threshold to obtain a light state, wherein the first threshold is greater than the second threshold.
4. The image signal processing method according to claim 3, It is characterized in that The step of performing a comparison operation on the light level value with the first threshold and the second threshold to obtain a light state comprises: When the preset threshold is greater than or equal to the first threshold, the lighting state is determined to be a poor lighting state.
5. The image signal processing method according to claim 4, It is characterized in that The step of controlling the opening or closing of a plurality of image processing modules according to the illumination state to obtain a processed image comprises: After determining the poor illumination state, a first control instruction is sent to the plurality of image processing modules to control the plurality of image processing modules to be in an on state.
6. The image signal processing method according to claim 3, It is characterized in that The step of performing a comparison operation on the light level value with the first threshold and the second threshold to obtain a light state comprises: When the preset threshold is greater than or equal to the second threshold and less than the first threshold, it is determined that the lighting state is a normal lighting state.
7. The image signal processing method according to claim 6, It is characterized in that The step of controlling the opening or closing of a plurality of image processing modules according to the illumination state to obtain a processed image comprises: After determining the general state of illumination, a second control instruction is sent to the plurality of image processing modules to control the image noise reduction unit and the color interpolation unit to be in an on state, and to control the image enhancement unit to be in a off state.
8. The image signal processing method according to claim 3, It is characterized in that The step of performing a comparison operation on the light level value with the first threshold and the second threshold to obtain a light state comprises: When the preset threshold is less than or equal to the second threshold, it is determined that the lighting state is a good lighting state.
9. The image signal processing method according to claim 8, It is characterized in that The step of controlling the opening or closing of a plurality of image processing modules according to the illumination state to obtain a processed image comprises: After determining that the illumination is in a good state, a third control instruction is sent to the plurality of image processing modules to control the color interpolation unit to be in an on state, and to control the image noise reduction unit and the image enhancement unit to be in an off state.
10. An image processing device, It is characterized in that include A data acquisition module, configured to acquire exposure parameters of the image sensor corresponding to the real-time image after the image sensor acquires image data of the current time in real time to acquire the real-time image data; A calculation module, used for determining the light level value at the current time according to the exposure parameter; A state determination module, used for obtaining the light state according to the light level value and a preset threshold; A control module, used for controlling the opening or closing of a plurality of image processing modules according to the illumination state to obtain a processed image; A transmission module is used to transmit the processed image to the automatic driving system.
11. The image processing device according to claim 10, It is characterized in that The image processing module includes an image noise reduction unit, a color interpolation unit and an image enhancement unit.
12. A device, It is characterized in that The method comprises a memory and a processor, wherein a program is stored in the processor, and when the program is executed by the processor, the image signal processing method according to any one of claims 1 to 9 is implemented.
13. A storage medium, It is characterized in that The storage medium stores a program, and when the program is executed by the processor, the image signal processing method according to any one of claims 1 to 9 is implemented.
