Image processing method and apparatus, electronic device, and medium

Abstract

The present application relates to the technical field of image processing, and discloses an image processing method and apparatus, an electronic device, and a medium. The image processing method in embodiments of the present application comprises: a first image signal processor performing abnormal frame detection processing on an image frame transmitted by an image sensor to obtain a detection result; when the detection result indicates that the image frame is an abnormal image frame, the first image signal processor performing repair processing on the abnormal image frame to obtain a repaired image frame; and transmitting the repaired image frame to a second image signal processor.

Description

Image processing method and device, electronic device and medium

[0001] Cross-reference to Related Applications

[0002] The present application claims priority to Chinese Patent Application No. 202411344709.3, filed on September 25, 2024, the contents of which are incorporated herein by reference in its entirety. TECHNICAL FIELD

[0003] The present application belongs to the technical field of image processing, and specifically relates to an image processing method, device, electronic device and medium. BACKGROUND

[0004] In a Mobile Industry Processor Interface (MIPI) based image transmission system, when transmitting image data between an image sensor and an externally connected pre-processing Image Signal Processor (ISP) chip (hereinafter referred to as an externally connected ISP chip), the image data may be disturbed by electromagnetic noise, signal crosstalk and physical connection defects, etc., resulting in data damage. Damaged image data can cause system lag, system crash and other problems, making it difficult to ensure reliable transmission of image data. SUMMARY

[0005] The purpose of the embodiments of the present application is to provide an image processing method, device, electronic device and medium, which can ensure the reliability of the transmission of abnormal image frames from the image sensor to the image signal processing module in the application processor, effectively avoiding system lag or system crash and other phenomena.

[0006] In a first aspect, the embodiments of the present application provide an image processing method, comprising:

[0007] The first image signal processor performs abnormal frame detection processing on the image frames transmitted by the image sensor to obtain a detection result;

[0008] The first image signal processor performs repair processing on the abnormal image frames to obtain repaired image frames in a case where the detection result indicates that the image frames are abnormal image frames;

[0009] The repaired image frames are transmitted to a second image signal processor.

[0010] In a second aspect, an image processing device is provided, comprising a first image signal processor, and the device comprises:

[0011] The processing module is configured to perform an abnormal frame detection process on an image frame transmitted by the image sensor to obtain a detection result; and perform a repair process on the abnormal image frame to obtain a repaired image frame when the detection result indicates that the image frame is an abnormal image frame.

[0012] The sending module is configured to transmit the repaired image frame to a second image signal processor.

[0013] In a third aspect, an electronic device is provided. The electronic device includes a processor and a memory. The memory stores programs or instructions executable on the processor. The programs or instructions, when executed by the processor, implement the steps of the method of the first aspect.

[0014] In a fourth aspect, a readable storage medium is provided. The readable storage medium stores programs or instructions. The programs or instructions, when executed by a processor, implement the steps of the method of the first aspect.

[0015] In a fifth aspect, a chip is provided. The chip includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is configured to execute programs or instructions to implement the method of the first aspect.

[0016] In a sixth aspect, a computer program product is provided. The program product is stored in a storage medium. The program product is executed by at least one processor to implement the method of the first aspect.

[0017] In the embodiments of the present application, the first image signal processor performs an abnormal frame detection process on an image frame transmitted by the image sensor to obtain a detection result; the first image signal processor performs a repair process on the abnormal image frame to obtain a repaired image frame when the detection result indicates that the image frame is an abnormal image frame; and the repaired image frame is transmitted to a second image signal processor. In this way, damaged image frames or incomplete image frames in the image transmission process can be identified in time, and the identified abnormal image frames can be repaired to ensure the integrity of the abnormal image frames, thereby ensuring the reliable transmission of the abnormal image frames in the transmission process from the image sensor to the image signal processing module in the application processor, and effectively avoiding phenomena such as system freezing or system crash. BRIEF DESCRIPTION OF DRAWINGS

[0018] FIG. 1 is a flowchart of an image processing method according to some embodiments of the present application;

[0019] FIG. 2 is a block diagram of an image processing apparatus according to some embodiments of the present application;

[0020] FIG. 3 is a structural block diagram of an electronic device according to some embodiments of the present application.

[0021] FIG. 4 is a structural block diagram of an electronic device according to some embodiments of the present application. DETAILED DESCRIPTION

[0022] The technical solutions in the embodiments of the present application will be clearly described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some, but not all, of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art belong to the scope of protection of the present application.

[0023] The terms "first", "second", and the like in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than that illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of a kind and do not limit the number of objects, for example, the first object can be one or more. In addition, "and / or" in the specification and claims means at least one of the connected objects, and the character " / ", generally represents a "or" relationship between the front and rear associated objects.

[0024] In order for those skilled in the art to better understand the embodiments of the present application, first, the following is explained.

[0025] It should be noted that the image processing method provided by the embodiments of the present application can be executed by a mobile phone, a tablet computer, a notebook computer, a palm computer, a vehicle-mounted electronic device, or the like. In some embodiments of the present application, the image processing method is executed by an electronic device as an execution subject, and the image processing method provided by the embodiments of the present application is described.

[0026] The image processing method provided by the embodiments of the present application can be applied to a camera image shooting scene, wherein one specific application scenario is a photograph preview scene, and another specific application scenario is a video recording scene.

[0027] The image processing method provided by the embodiments of the present application will be described in detail below in combination with the drawings and specific embodiments and their application scenarios.

[0028] As shown in FIG. 1, the present application provides an image processing method, which comprises:

[0029] Step 101: The first image signal processor performs abnormal frame detection processing on the image frame transmitted by the image sensor to obtain a detection result.

[0030] Optionally, the first image signal processor has an image preprocessing function, the first image signal processor is connected with the image sensor and the second image signal processor respectively, and the second image signal processor is an image signal processing module in the application processor.

[0031] In the embodiment of the application, the first image signal processor is an image signal processing (ISP) chip. The image preprocessing function includes, but is not limited to, initial denoising, color correction, and basic enhancement processing. Optionally, the abnormal image frame includes at least one of the following: an image frame with damaged pixel rows and an image frame with missing pixel rows.

[0032] In this step, the first image signal processor performs real-time abnormal frame detection processing on the image frame transmitted by the image sensor, so as to determine whether there is an abnormal image frame in the image frame transmitted by the image sensor.

[0033] For example, in a photographing preview scene, the image sensor collects a preview image frame and transmits it to the first image signal processor. If the first image signal processor detects that at least one pixel row in the preview image frame is damaged, it is determined that the preview image frame is an abnormal image frame.

[0034] For another example, in a video recording scene, the image sensor transmits the image frame of the recorded video to the first image signal processor in real time. If the first image signal processor detects that at least one pixel row of the image frame is missing, it is determined that the image frame is an abnormal image frame.

[0035] Step 102: In the case where the detection result indicates that the image frame is an abnormal image frame, the first image signal processor performs repair processing on the abnormal image frame to obtain a repaired image frame.

[0036] Optionally, in the case where the detection result indicates that the image frame is an abnormal image frame, the first image signal processor does not perform image preprocessing on the abnormal image frame, and performs repair processing on the abnormal image frame to obtain a repaired image frame.

[0037] Optionally, the repair processing is used to ensure the integrity of the abnormal image frame.

[0038] Optionally, in the case where the detection result indicates that the image frame is not an abnormal image frame, the image frame is subjected to image preprocessing.

[0039] In this step, the abnormal image frame is not subjected to image preprocessing, so as to avoid the influence of the image content of the abnormal image frame on the image quality.

[0040] For example, in a photograph preview scene, if the detection result indicates that at least one pixel row in a preview image frame is damaged, the damaged pixel row of the preview image frame is repaired to complete the image content of the damaged pixel row, and a repaired image frame is obtained.

[0041] For example, in a photograph preview scene, if the detection result indicates that at least one pixel row in a preview image frame is damaged, the damaged pixel row of the preview image frame is repaired to complete the image content of the damaged pixel row, and a repaired image frame is obtained.

[0042] Step 103: transmitting the repaired image frame to a second image signal processor.

[0043] For example, in a photograph preview scene, the repaired preview image frame is transmitted to the second image signal processor; for example, in a video recording scene, the repaired image frame of the video is transmitted to the second image signal processor.

[0044] Here, the repaired image frame is transmitted to the second image signal processor to ensure reliable transmission of image data, and the second image signal processor takes corresponding abnormal processing strategies after receiving the repaired image frame.

[0045] In the embodiments of the present application, the first image signal processor performs abnormal frame detection processing on the image frame transmitted by the image sensor to obtain a detection result; the first image signal processor performs repair processing on the abnormal image frame to obtain a repaired image frame when the detection result indicates that the image frame is an abnormal image frame; and the repaired image frame is transmitted to a second image signal processor. Thus, the damaged image frame or the incomplete image frame in the image transmission process can be identified in time, and the identified abnormal image frame can be repaired to ensure the integrity of the abnormal image frame, thereby ensuring the reliable transmission of the abnormal image frame from the image sensor to the image signal processing module in the application processor, and effectively avoiding the phenomenon of system freezing or system crash.

[0046] Optionally, the first image signal processor performs abnormal frame detection processing on the image frame transmitted by the image sensor to obtain a detection result, including:

[0047] Obtaining a standard image parameter corresponding to a sensor parameter of the image sensor, the standard image parameter including at least one of a standard number of rows of image frames and a standard row interval, and the sensor parameter including at least one of a configuration parameter of the sensor and an operation mode parameter of the sensor;

[0048] According to the relationship between the parameter difference and the parameter difference threshold, a detection result of the image frame is determined, the parameter difference being a difference between a parameter value of an image parameter of an image frame output by the image sensor and a parameter value of the standard image parameter.

[0049] Optionally, the configuration parameters of the above-mentioned sensor include but are not limited to pixel size, resolution, and frame rate, etc. The operation mode parameters of the above-mentioned sensor include shooting mode, for example, the shooting mode can be general shooting mode or high dynamic range shooting mode or preview mode.

[0050] In some embodiments of the present application, the standard image parameters are configured based on the sensor parameters of the image sensor, and different sensor parameters correspond to different standard image parameters. For example, different sensor parameters correspond to different number of rows and row spacing.

[0051] As an implementation manner, the standard image parameters corresponding to the sensor parameters of the image sensor are obtained based on the technical manual of the image sensor or the technical specification document provided by the sensor manufacturer.

[0052] For example, in the photograph preview scene, the standard number of rows A1 of the image frame corresponding to the sensor parameters of the image sensor, and the standard row spacing B1 of the image frame corresponding to the sensor parameters of the image sensor;

[0053] In the video recording scene, the standard number of rows A2 of the image frame corresponding to the sensor parameters of the image sensor, and the standard row spacing B2 of the image frame corresponding to the sensor parameters of the image sensor, wherein A1 is different from A2, and B1 is different from B2.

[0054] Optionally, as an implementation manner, the detection result of the image frame is determined according to the relationship between the parameter difference value and the parameter difference value threshold, including: if the parameter difference value is greater than the parameter difference value threshold, determining that the image frame is an abnormal image frame, otherwise, determining that the image frame is a normal image frame.

[0055] In some embodiments of the present application, the image parameters of the image frame transmitted by the image sensor are compared with the standard image parameters; if the difference value between the image parameters of the image frame transmitted by the image sensor and the standard image parameters is greater than a preset threshold, it is determined that the image frame is an abnormal image frame. For example, if the number of rows of the image frame transmitted by the image sensor is 260 rows, the pixel row spacing is 15 pixels, the standard number of rows is 300 rows, the standard pixel row spacing is 30 pixels, the row number difference threshold is 10, and the pixel row spacing threshold is 10, the row number difference between the number of rows of the image frame and the standard number of rows is 40 rows, which is greater than the row number difference threshold, and the difference between the pixel row spacing of the image frame and the standard pixel row spacing is 15 pixels, which is greater than the pixel row spacing threshold, therefore, it is determined that the image frame is an abnormal image frame.

[0056] In a case where a difference between a value of an image parameter of an image frame transmitted by the image sensor and a value of a standard image parameter is greater than a parameter difference threshold, it is indicated that a deviation between the two exceeds a preset deviation, and it is determined that the image frame is an abnormal image frame. The parameter difference threshold can be set based on a stable operation performance of the image sensor and a minimum error allowed.

[0057] In some embodiments of the present application, whether each image frame transmitted by the image sensor is an abnormal image frame is monitored based on a real-time monitoring algorithm, and a special circuit capable of executing the real-time monitoring algorithm is designed according to specifications of the image sensor and a configuration selected, to ensure that the special circuit can analyze the image frame arriving in time, so that the abnormal frame can be quickly identified when the abnormal frame appears, to ensure the integrity and reliability of image transmission in the process of image transmission based on the MIPI interface.

[0058] Optionally, the repairing processing of the abnormal image frame includes:

[0059] identifying an abnormal pixel row in the abnormal image frame, the abnormal pixel row being a missing or damaged row in the abnormal image frame;

[0060] performing repairing processing on image data of the abnormal pixel row in the abnormal image frame according to reference image data, to obtain a repaired image frame;

[0061] The reference image data includes image data of the abnormal pixel row of a reference image frame, or image data of a reference pixel row of the abnormal image frame, a frame interval between the reference image frame and the abnormal image frame is less than a frame interval threshold, and a row interval between the reference pixel row and the abnormal pixel row is less than a row interval threshold.

[0062] In some embodiments of the present application, the image data is image content of an image frame.

[0063] As an implementation manner, the reference image frame is an adjacent image frame of the abnormal image frame, for example, the reference image frame is a previous image frame or a next image frame of the abnormal image frame. The image data of the abnormal image frame is complemented based on image data of the adjacent image frame, to ensure continuity of image output, thereby effectively ensuring reliability of image transmission.

[0064] As an implementation manner, the reference pixel row is an adjacent pixel row of the abnormal pixel row.

[0065] For example, in a case where image data of an abnormal pixel row of the reference image frame is unavailable, the image data of the abnormal pixel row is generated based on image data of an adjacent pixel row of the abnormal pixel row of the abnormal image frame, using a difference algorithm.

[0066] For example, in a photograph preview scenario, if the 10th-15th rows of an abnormal preview image frame are abnormal pixel rows, the image data of the 10th-15th rows can be generated based on the image data of the 5th-9th rows of the abnormal preview image frame.

[0067] For example, in a video recording scenario, if the 1st-5th rows of a third image frame of a recorded video are abnormal pixel rows, the image data of the 1st-5th rows of the third image frame can be generated based on the image data of the 1st-5th rows of a second image frame of the recorded video.

[0068] Optionally, the image data of the abnormal pixel rows in the abnormal image frame is repaired, including:

[0069] In a case where the number of the abnormal pixel rows is less than a pixel row threshold, the image data of the abnormal pixel rows in the abnormal image frame is repaired according to reference image data.

[0070] In a case where the number of the abnormal pixel rows is greater than or equal to the pixel row threshold, at least one of the image sensor, the first image signal processor and the second image signal processor is restarted or reset.

[0071] In a case where the number of the abnormal pixel rows is less than the pixel row threshold, it is indicated that the abnormal image frame is image damage caused by abnormal interference rather than image damage caused by device abnormality, and thus the image data of a target row in the abnormal image frame can be repaired, so as to ensure reliable transmission of the image data.

[0072] In the embodiment of the application, the abnormal pixel rows in the abnormal image frame are complemented according to the reference image data, so as to ensure continuity of image output, thereby effectively ensuring reliability of image transmission.

[0073] Optionally, the repairing of the abnormal image frame includes:

[0074] In the buffered image frames, an image frame with a time stamp difference from the time stamp of the abnormal image frame less than a time stamp threshold is selected as a repaired image frame.

[0075] Optionally, the time stamp can be understood as a start scanning time of the image frame.

[0076] As an implementation manner, in the buffered image frames, an image frame with a minimum time stamp difference from the time stamp of the abnormal image frame is selected as the repaired image frame.

[0077] Optionally, in the embodiments of the present application, the first image signal processor caches a preset number of image frames and continuously updates the cached image frames, wherein the cached image frames do not include abnormal image frames, i.e., the cached image frames are valid image frames. For example, the first image signal processor caches 3 image frames, i.e., the first image frame to the third image frame. After receiving the fourth image frame, the cached image frames are updated to the second image frame to the fourth image frame.

[0078] For example, the first image signal processor caches three image frames, i.e., the image frame S1, the image frame S2 and the image frame S3. The time stamps of the three image frames are T1, T2 and T3 respectively, wherein the time corresponding to T1 is earlier than the time corresponding to T2, and the time corresponding to T2 is earlier than the time corresponding to T3. After receiving the image frame S4, the first image signal processor detects that the image frame S4 is an abnormal image frame. Then, the time stamp T4 of the image frame S4 is compared with T1, T2 and T3 respectively, and it is determined that T3 is closest to T4. Therefore, the image frame S4 is taken as a repair image frame.

[0079] In the embodiments of the present application, in order to avoid the interruption of the image caused by the frame loss, the first image signal processor continuously caches a certain number of valid image frames inside. When an abnormal image frame is detected, an image frame with a time stamp difference less than a time stamp threshold is selected from the cached image frames as a repair image frame, so as to ensure the continuity and stability of the image output.

[0080] Optionally, the method of the embodiments of the present application further comprises:

[0081] The first image signal processor obtains indication information of the image frame transmitted by the image sensor according to the reference information, and the indication information is used to indicate whether the image frame is an abnormal image frame.

[0082] The indication information is transmitted to the second image signal processor.

[0083] The reference information includes at least one of the following: control data bits of the image frame, image content indication information of the image frame, and error flag bits.

[0084] In some embodiments of the present application, the data at a specific position of the content transmitted by the first image signal processor to the second image signal processor is determined as a control data bit, for example, the first piece of data or the last piece of data of the transmitted content is determined as a control data bit, and the data corresponding to the control data bit does not include image content of the image frame. For example, the control data bit corresponding to a normal image frame is data 1, and the control data bit corresponding to an abnormal image frame is data 2.

[0085] In some embodiments of the present application, the image content indication information can be data information of a specific position in the image content corresponding to the image frame. For example, if the data information of the specific position in the image content corresponding to the image frame is all 0, it is determined that the image frame is an abnormal image frame.

[0086] In some embodiments of the present application, the error flag can be an error code in a log or image metadata, through which the application processor can correctly distinguish between normal image frames and abnormal image frames. For example, if the error flag is 1, it indicates that the image frame is a normal image frame, and if the error flag is 0, it indicates that the image frame is an abnormal image frame.

[0087] In addition, through the above-mentioned log and image metadata, an error code containing detailed abnormal reason and condition can also be delivered to the application processor, and relevant frame number and timestamp can also be provided to facilitate the application processor to record the log and track the problem later.

[0088] In the embodiments of the present application, when an abnormal image frame is detected, the indication information indicating whether the image frame is an abnormal image frame is obtained through the reference information and transmitted to the second image signal processing, so that the second image signal processor can identify the abnormal image frame. In addition, the first image signal processor filters out the abnormal image frame from the normal processing flow, that is, does not perform image preprocessing on the abnormal image frame, so as to avoid bad influence on the image quality.

[0089] Optionally, the method of the embodiments of the present application further comprises:

[0090] The first image signal processor sends an interrupt signal to the second image signal processor, and the interrupt signal is used to trigger the identification process of the abnormal image frame.

[0091] The interrupt signal can be a hardware interrupt request signal (Interrupt Request, IRQ), and the interrupt type corresponding to the interrupt request signal includes but is not limited to: abnormal data packet, missing data line, data packet timeout, etc.

[0092] Illustratively, after receiving the interrupt signal, the second image signal processor reads the above-mentioned indication information through the corresponding driver to determine whether the image frame is an abnormal image frame. The second image signal processor will execute different response measures according to the type and severity of the abnormality. This can include: ignoring the abnormal frame with smaller defects, restoring the sensor settings, or restarting the hardware and software system. In addition, the second image signal processor can record all abnormal events into the system log for subsequent analysis, so as to improve the robustness of the system and assist future error-proof design.

[0093] In the embodiments of the present application, when the first image signal processor detects an abnormal image frame, an interrupt signal is sent so that the second signal processor identifies the abnormal image frame based on the interrupt signal, and then takes a corresponding processing strategy for the abnormal image frame.

[0094] Through the above scheme, the reliability and stability of the external ISP chip in actual application can be significantly improved. Through the internal abnormal processing mechanism, the influence of abnormal frames on the final image quality is reduced. These measures not only guarantee the integrity of image data, but also provide flexibility and adaptability for operation under abnormal conditions.

[0095] Optionally, the method of the embodiments of the present application further comprises:

[0096] When the number of abnormal image frames in the image frames transmitted by the image sensor is greater than the frame number threshold, the first image signal processor performs at least one of the following:

[0097] Performing a reset process, wherein the reset process includes at least one of the following: reinitializing the interface of the image sensor, resetting the configuration of the image sensor;

[0098] Outputting alarm information.

[0099] The above-mentioned resetting the configuration of the image sensor can be understood as resetting the image sensor.

[0100] In some embodiments of the present application, the configuration of the image sensor is used to configure the image sensor to collect image data, and the configuration of the image sensor includes but is not limited to resolution, frame rate, image format, etc.

[0101] By taking the above-mentioned reset processing strategy to ensure that the image sensor returns to the normal working state, of course, the reset processing can also include other operations that can make the image sensor return to the normal working state, which is not limited here. By outputting the alarm information, the external control system can take additional intervention measures, such as switching to a backup sensor or performing system-level error detection and repair.

[0102] Optionally, the first image signal processor is connected with the image sensor and the second image signal processor respectively, and the first image signal processor has an image preprocessing function.

[0103] The second image signal processor is an image signal processing module in an application processor (Application Processor, AP).

[0104] The above-mentioned first image signal processor can also be described as an external ISP.

[0105] The image preprocessing function of the first image signal processor has been described in the above description, and will not be repeated here.

[0106] In the embodiments of the present application, the first image signal processor with image preprocessing function is arranged between the image sensor and the second image signal processor. In addition to the image preprocessing function, the first image signal processor can also perform abnormal frame detection and abnormal frame repair processing on the image frames transmitted by the image sensor, thereby ensuring the reliable transmission of abnormal image frames in the transmission process from the image sensor to the image signal processing module in the application processor, effectively avoiding the phenomenon of system freezing or system crash.

[0107] In the embodiments of the present application, a comprehensive and reliable abnormal information transmission and processing mechanism is established between the external ISP and the AP. This mechanism can identify multiple types of abnormalities and provide corresponding abnormal processing strategies for each type. Through this scheme, the system can efficiently handle abnormal situations, reduce the impact of abnormal events on the end-user experience, and thus improve the stability and reliability of the entire image system.

[0108] It should be noted that the scheme of the present application has wide applicability and expandability, not only suitable for the current mobile phone imaging field, but also suitable for the following possible application scenarios:

[0109] Medical imaging system: In medical imaging devices such as X-ray machines, computed tomography (CT), and other high-precision imaging systems, image quality directly affects the accuracy of diagnostic results. It is essential to apply the scheme of the present application to ensure the integrity and reliability of imaging data.

[0110] Vehicle-mounted camera system: In automatic driving and advanced driving assistance systems (ADAS), vehicle-mounted cameras provide critical visual information. Rapid detection and correction of abnormal frames can improve road safety and reduce risks caused by picture delays or errors.

[0111] Security monitoring: Security camera systems require 24 / 7 uninterrupted operation and extremely high data reliability. The application of the present application can help the system to timely discover and handle abnormal pictures and ensure the continuity of monitoring data.

[0112] Motion capture and analysis: In sports competition or animation production motion capture, distorted images can affect the accuracy of motion data analysis. Early identification and processing of abnormal frames can provide more stable and reliable data output.

[0113] Aerospace remote sensing: In the imaging system carried by satellite or unmanned aerial vehicle, high-quality image data is essential for the collection and analysis of geographic information. The application can ensure that the quality of image data is maintained as much as possible even in adverse weather or signal interference conditions.

[0114] Industrial vision detection system: In the automated production line, image detection is used to monitor product quality. The detection and processing of abnormal frames can correct image data errors in real time and ensure the consistency of product quality.

[0115] The image processing method provided by the embodiment of the application can be executed by an image processing device. In the embodiment of the application, the image processing device is taken as an example to illustrate the image processing device provided by the embodiment of the application.

[0116] As shown in FIG. 2, the embodiment of the application further provides an image processing device 200, which comprises a first image signal processor, and the device comprises:

[0117] The processing module 201 is configured to perform abnormal frame detection processing on the image frame transmitted by the image sensor to obtain a detection result, and perform repair processing on the abnormal image frame to obtain a repaired image frame in a case where the detection result indicates that the image frame is an abnormal image frame.

[0118] The sending module 202 is configured to transmit the repaired image frame to a second image signal processor.

[0119] Optionally, the processing module is specifically configured to:

[0120] obtain a standard image parameter corresponding to a sensor parameter of the image sensor, the standard image parameter comprising at least one of a standard number of rows of image frames and a standard row interval, and the sensor parameter comprising at least one of a configuration parameter of the sensor and an operation mode parameter of the sensor.

[0121] According to the relationship between the parameter difference and the parameter difference threshold, the detection result of the image frame is determined, and the parameter difference is the difference between the parameter value of the image parameter of the image frame output by the image sensor and the parameter value of the standard image parameter.

[0122] Optionally, the processing module is specifically configured to:

[0123] identify an abnormal pixel row in the abnormal image frame, the abnormal pixel row being a missing or damaged row in the abnormal image frame.

[0124] According to the reference image data, the image data of the abnormal pixel row in the abnormal image frame is repaired to obtain a repaired image frame.

[0125] The reference image data includes image data of the abnormal pixel row of a reference image frame, or image data of a reference pixel row of the abnormal image frame, the reference image frame has a frame interval less than a frame interval threshold value from the abnormal image frame, and the reference pixel row has a row interval less than a row interval threshold value from the abnormal pixel row.

[0126] Optionally, the processing module is specifically configured to:

[0127] In the buffered image frames, an image frame having a time stamp difference less than a time stamp threshold value from the time stamp of the abnormal image frame is selected as a repair image frame.

[0128] Optionally, the processing module is further configured to:

[0129] According to the reference information, the indication information of the image frame transmitted by the image sensor is obtained, and the indication information is used to indicate whether the image frame is an abnormal image frame.

[0130] The indication information is sent to the second image signal processor.

[0131] The reference information includes at least one of the following: control data bits of the image frame, image content indication information of the image frame, and an error flag bit.

[0132] Optionally, the sending module is further configured to:

[0133] An interrupt signal is sent to the second image signal processor, and the interrupt signal is used to trigger an identification process of the abnormal image frame.

[0134] Optionally, the processing module is further configured to:

[0135] In a case where the number of abnormal image frames in the image frames transmitted by the image sensor is greater than a frame number threshold value, at least one of the following is performed:

[0136] Reset processing is performed, and the reset processing includes at least one of the following: reinitializing an interface of the image sensor, and resetting a configuration of the image sensor.

[0137] Alarm information is output.

[0138] Optionally, the first image signal processor is connected with the image sensor and the second image signal processor respectively, and the first image signal processor has an image preprocessing function.

[0139] The second image signal processor is an image signal processing module in an application processor.

[0140] In the embodiment of the present application, the first image signal processor performs abnormal frame detection processing on the image frames transmitted by the image sensor to obtain a detection result; the first image signal processor performs repair processing on the abnormal image frames in a case where the detection result indicates that the image frames are abnormal image frames to obtain repaired image frames; and the repaired image frames are transmitted to the second image signal processor. In this way, the damaged image frames or incomplete image frames in the image transmission process can be identified in time, and the identified abnormal image frames can be repaired to ensure the integrity of the abnormal image frames, thereby ensuring the reliable transmission of the abnormal image frames in the transmission process from the image sensor to the image signal processing module in the application processor, and effectively avoiding the phenomenon of system freezing or system crash.

[0141] The image processing apparatus in the embodiment of the present application can be an electronic device or a component in an electronic device, for example, an integrated circuit or a chip. The electronic device can be a terminal or other devices other than a terminal. For example, the electronic device can be a mobile phone, a tablet computer, a notebook computer, a palm computer, a vehicle-mounted electronic device, a mobile Internet device (MID), an augmented reality (AR) / virtual reality (VR) device, a robot, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), and can also be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine, or a self-service machine, and the like, and the embodiment of the present application is not limited in this regard.

[0142] The image processing apparatus in the embodiment of the present application can be a device with an operating system. The operating system can be an Android operating system, an ios operating system, or other possible operating systems, and the embodiment of the present application is not limited in this regard.

[0143] The image processing apparatus provided in the embodiment of the present application can implement each process implemented by the method embodiment of FIG. 1, and thus details are not repeated here.

[0144] Optionally, as shown in FIG. 3, the embodiment of the present application further provides an electronic device 300, comprising a processor 301 and a memory 302, wherein the memory 302 stores programs or instructions executable on the processor 301, and the programs or instructions are executed by the processor 301 to implement each step of the image processing method embodiment described above and achieve the same technical effects. To avoid repetition, details are not described herein.

[0145] It should be noted that the electronic device in the embodiment of the present application includes the mobile electronic device and the non-mobile electronic device described above.

[0146] FIG. 4 is a schematic diagram of a hardware structure of an electronic device implementing the embodiment of the present application.

[0147] The electronic device 400 includes, but is not limited to, a radio frequency unit 401, a network module 402, an audio output unit 403, an input unit 404, a sensor 405, a display unit 406, a user input unit 407, an interface unit 408, a memory 409, and a processor 410, etc.

[0148] Those skilled in the art can understand that the electronic device 400 can further include a power supply (such as a battery) for supplying power to each component, and the power supply can be logically connected to the processor 410 through a power management system, so as to realize functions such as management of charging, discharging, and power consumption management through the power management system. The electronic device structure shown in FIG. 4 does not constitute a limitation on the electronic device, and the electronic device can include more or fewer components than shown, or combine certain components, or different component arrangements, which are not described herein.

[0149] The processor 410 is configured to perform abnormal frame detection processing on an image frame transmitted by an image sensor to obtain a detection result, perform repair processing on the abnormal image frame to obtain a repaired image frame in a case where the detection result indicates that the image frame is an abnormal image frame, and the radio frequency unit 401 is configured to transmit the repaired image frame to a second image signal processor.

[0150] Optionally, the processor 410 is specifically configured to:

[0151] obtain a standard image parameter corresponding to a sensor parameter of the image sensor, the standard image parameter including at least one of a standard number of rows of image frames and a standard row interval, and the sensor parameter including at least one of a configuration parameter of the sensor and an operation mode parameter of the sensor;

[0152] determine a detection result of the image frame according to a relationship between a parameter difference value and a parameter difference threshold value, the parameter difference value being a difference between a parameter value of an image parameter of an image frame output by the image sensor and a parameter value of the standard image parameter.

[0153] Optionally, the processor 410 is specifically configured to:

[0154] identify an abnormal pixel row in the abnormal image frame, the abnormal pixel row being a missing or damaged row in the abnormal image frame;

[0155] perform a repair process on image data of the abnormal pixel row in the abnormal image frame according to reference image data, to obtain a repaired image frame;

[0156] The reference image data includes: image data of the abnormal pixel row of a reference image frame; or image data of a reference pixel row of the abnormal image frame, the reference image frame having a frame interval less than a frame interval threshold value from the abnormal image frame, and the reference pixel row having a row interval less than a row interval threshold value from the abnormal pixel row.

[0157] Optionally, the processor 410 is specifically configured to:

[0158] select, from the buffered image frames, an image frame having a time stamp difference less than a time stamp threshold value from a time stamp of the abnormal image frame, as the repaired image frame.

[0159] Optionally, the processor 410 is further configured to:

[0160] The first image signal processor obtains, according to reference information, indication information of an image frame transmitted by the image sensor, the indication information being used to indicate whether the image frame is an abnormal image frame;

[0161] transmit the indication information to the second image signal processor;

[0162] The reference information includes at least one of the following: control data bits of the image frame, image content indication information of the image frame, and an error flag bit.

[0163] Optionally, the radio frequency unit 401 is further configured to:

[0164] The first image signal processor transmits an interrupt signal to the second image signal processor, the interrupt signal being used to trigger an identification process of an abnormal image frame.

[0165] Optionally, the processor 410 is further configured to:

[0166] When the number of abnormal image frames in the image frames transmitted by the image sensor is greater than a frame number threshold value, the first image signal processor performs at least one of the following:

[0167] perform a reset process, wherein the reset process includes at least one of the following: reinitializing an interface of the image sensor, and resetting a configuration of the image sensor;

[0168] outputting the alarm information.

[0169] Optionally, the first image signal processor is connected with the image sensor and the second image signal processor respectively, and the first image signal processor has an image preprocessing function.

[0170] The second image signal processor is an image signal processing module in the application processor.

[0171] In the embodiment of the application, the first image signal processor performs abnormal frame detection processing on the image frame transmitted by the image sensor to obtain a detection result; the first image signal processor performs repair processing on the abnormal image frame in a case where the detection result indicates that the image frame is an abnormal image frame to obtain a repaired image frame; and the repaired image frame is transmitted to the second image signal processor. In this way, the damaged image frame or the incomplete image frame in the image transmission process can be identified in time, and the identified abnormal image frame can be repaired to ensure the integrity of the abnormal image frame, thereby ensuring the reliable transmission of the abnormal image frame in the transmission process from the image sensor to the image signal processing module in the application processor, and effectively avoiding the phenomenon of system freezing or system crash.

[0172] It should be understood that, in the embodiment of the application, the input unit 404 can include a graphics processor (GPU) 4041 and a microphone 4042. The graphics processor 4041 processes image data of a still picture or a video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 406 can include a display panel 4061, which can be configured in the form of a liquid crystal display, an organic light-emitting diode, etc. The user input unit 407 includes at least one of a touch panel 4071 and other input devices 4072. The touch panel 4071 is also called a touch screen. The touch panel 4071 can include a touch detection device and a touch controller. The other input devices 4072 can include, but are not limited to, a physical keyboard, function keys (such as volume control keys, on-off keys, etc.), a trackball, a mouse, an operating lever, and the like, which will not be described here.

[0173] The memory 409 can be used to store software programs and various data. The memory 409 can mainly include a first storage area storing programs or instructions and a second storage area storing data, wherein the first storage area can store an operating system, application programs or instructions required by at least one function (such as a sound playing function, an image playing function, etc.), and the like. In addition, the memory 409 can include a volatile memory or a non-volatile memory, or the memory 409 can include both a volatile memory and a non-volatile memory. The non-volatile memory can be a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically EPROM (EEPROM), or a flash memory. The volatile memory can be a Random Access Memory (RAM), a Static RAM (SRAM), a Dynamic RAM (DRAM), a Synchronous DRAM (SDRAM), a Double Data Rate SDRAM (DDR SDRAM), an Enhanced SDRAM (ESDRAM), a Synch link DRAM (SLDRAM), and a Direct Rambus RAM (DRRAM). The memory 409 in the embodiments of the present application includes but is not limited to these and any other suitable types of memory.

[0174] The processor 410 can include one or more processing units; optionally, the processor 410 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and an application program, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 410.

[0175] The embodiments of the present application also provide a readable storage medium, the readable storage medium stores programs or instructions, the programs or instructions are executed by a processor to realize various processes of the above-mentioned image processing method embodiments, and the same technical effects can be achieved. To avoid repetition, details are not described here.

[0176] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes a computer readable storage medium, such as a computer readable only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0177] The embodiment of the present application further provides a chip, which comprises a processor and a communication interface, the communication interface is coupled with the processor, the processor is used for running programs or instructions to realize the processes of the above image processing method embodiments and achieve the same technical effects. To avoid repetition, details are not described herein.

[0178] It should be understood that the chip mentioned in the embodiment of the present application can also be referred to as a system level chip, a system chip, a chip system or a system on chip, etc.

[0179] The embodiment of the present application provides a computer program product, which is stored in a storage medium, and is executed by at least one processor to realize the processes of the above image processing method embodiments and achieve the same technical effects. To avoid repetition, details are not described herein.

[0180] It should be noted that in this document, the term "comprising" or "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the method and device in the embodiment of the present application is not limited to the order of performing the functions as shown or discussed, but can also include performing the functions in a substantially simultaneous manner or in the opposite order, for example, the described method can be performed in an order different from that described, and various steps can also be added, omitted or combined. In addition, the features described with reference to some examples can be combined in other examples.

[0181] Those skilled in the art can clearly understand the above-mentioned embodiment method can be realized by means of software and necessary general hardware platform, of course, also can be through hardware, but in many cases the former is the better embodiment. Based on such understanding, the technical solutions of the present application essentially or say the part of the related art to make contributions can be embodied in the form of computer software product, the computer software product is stored in a storage medium (such as ROM / RAM, disk, optical disc), including a number of instructions to make a terminal (may be a mobile phone, computer, server, or network equipment, etc.) executes the method described in various embodiments of the present application.

[0182] The embodiments of the present application are described above in combination with the drawings, but the present application is not limited to the above-mentioned specific embodiments, the above-mentioned specific embodiments are only illustrative, but not limited, those skilled in the art can make many forms without departing from the purpose of the present application and the scope protected by the claims under the inspiration of the present application, all belong to the protection of the present application.

Claims

1. An image processing method comprising: a first image signal processor performing an abnormal frame detection process on an image frame transmitted by an image sensor to obtain a detection result; in a case where the detection result indicates that the image frame is an abnormal image frame, the first image signal processor performing a repair process on the abnormal image frame to obtain a repaired image frame; transmitting the repaired image frame to a second image signal processor.

2. The method of claim 1, wherein, The first image signal processor performs an abnormal frame detection process on an image frame transmitted by an image sensor to obtain a detection result, comprising: obtaining a standard image parameter corresponding to a sensor parameter of the image sensor, the standard image parameter comprising at least one of the following of the image frame: a standard row number, a standard row interval, the sensor parameter comprising at least one of the following of the sensor: a configuration parameter of the sensor, an operating mode parameter of the sensor; determining the detection result of the image frame according to a relationship between a parameter difference value and a parameter difference value threshold, the parameter difference value being a difference value between a parameter value of an image parameter of an image frame output by the image sensor and a parameter value of the standard image parameter.

3. The method of claim 1, wherein, The repair process on the abnormal image frame to obtain a repaired image frame, comprising: identifying an abnormal pixel row in the abnormal image frame, the abnormal pixel row being a missing or damaged row in the abnormal image frame; performing a repair process on image data of the abnormal pixel row in the abnormal image frame according to reference image data to obtain a repaired image frame; wherein the reference image data comprises: image data of the abnormal pixel row of a reference image frame; or image data of a reference pixel row of the abnormal image frame, the frame interval between the reference image frame and the abnormal image frame being less than a frame interval threshold, and the row interval between the reference pixel row and the abnormal pixel row being less than a row interval threshold.

4. The method of claim 1, wherein, The repair process on the abnormal image frame to obtain a repaired image frame, comprising: selecting, from the buffered image frames, an image frame having a time stamp difference from the time stamp of the abnormal image frame less than a time stamp threshold as the repaired image frame.

5. The method of claim 1, wherein, The method further comprises: The first image signal processor obtains indication information of the image frame transmitted by the image sensor according to reference information, the indication information being used to indicate whether the image frame is an abnormal image frame; sending the indication information to the second image signal processor; wherein the reference information comprises at least one of the following: control data bits of the image frame, image content indication information of the image frame, an error flag bit.

6. The method of claim 1, wherein, The method further comprises: The first image signal processor sends an interrupt signal to the second image signal processor, the interrupt signal being used to trigger an identification process of an abnormal image frame.

7. The method of claim 1, wherein, The method further comprises: In a case where the number of abnormal image frames in the image frames transmitted by the image sensor is greater than a frame number threshold, the first image signal processor performs at least one of the following: performing a reset process, wherein the reset process comprises at least one of the following: reinitializing an interface of the image sensor, resetting a configuration of the image sensor; outputting alarm information.

8. The method of claim 1, wherein, The first image signal processor is connected with the image sensor and the second image signal processor respectively, and has an image preprocessing function. The second image signal processor is an image signal processing module in an application processor. 9.An image processing apparatus comprising a first image signal processor, the apparatus comprising: a processing module configured to perform an abnormal frame detection process on an image frame transmitted by an image sensor to obtain a detection result; in a case where the detection result indicates that the image frame is an abnormal image frame, performing a repair process on the abnormal image frame to obtain a repaired image frame; a sending module configured to transmit the repaired image frame to a second image signal processor.

10. The apparatus of claim 9, wherein, The processing module is specifically configured to: obtain a standard image parameter corresponding to a sensor parameter of the image sensor, the standard image parameter comprising at least one of a standard number of rows of an image frame and a standard row interval, and the sensor parameter comprising at least one of a configuration parameter of the image sensor and an operation mode parameter of the image sensor; determine the detection result of the image frame according to a relationship between a parameter difference value and a parameter difference threshold value, the parameter difference value being a difference between a parameter value of an image parameter of an image frame output by the image sensor and a parameter value of the standard image parameter.

11. The apparatus of claim 9, wherein, The processing module is specifically configured to: identify an abnormal pixel row in the abnormal image frame, the abnormal pixel row being a missing or damaged row in the abnormal image frame; perform a repair process on image data of the abnormal pixel row in the abnormal image frame according to reference image data to obtain a repaired image frame; wherein the reference image data comprises image data of the abnormal pixel row of a reference image frame, or image data of a reference pixel row of the abnormal image frame, a frame interval between the reference image frame and the abnormal image frame being less than a frame interval threshold value, and a row interval between the reference pixel row and the abnormal pixel row being less than a row interval threshold value.

12. The apparatus of claim 9, wherein, The processing module is specifically configured to: select, from the buffered image frames, an image frame having a time stamp difference from the time stamp of the abnormal image frame less than a time stamp threshold value as the repaired image frame.

13. The apparatus of claim 9, wherein, The processing module is further configured to: obtain, according to reference information, indication information of the image frame transmitted by the image sensor, the indication information being used to indicate whether the image frame is an abnormal image frame; send the indication information to the second image signal processor; wherein the reference information comprises at least one of a control data bit of the image frame, image content indication information of the image frame, and an error flag bit.

14. The apparatus of claim 9, wherein, The sending module is further configured to: send an interrupt signal to the second image signal processor, the interrupt signal being used to trigger an identification process of the abnormal image frame.

15. The apparatus of claim 9, wherein, The processing module is further configured to: in a case where a number of abnormal image frames in the image frames transmitted by the image sensor is greater than a frame number threshold value, perform at least one of the following: perform a reset process, wherein the reset process comprises at least one of reinitializing an interface of the image sensor and resetting a configuration of the image sensor; output alarm information.

16. The apparatus of claim 9, wherein, The first image signal processor is connected with the image sensor and the second image signal processor respectively, and has an image preprocessing function. The second image signal processor is an image signal processing module in an application processor. 17.An electronic device, comprising a processor and a memory, wherein the memory stores a program or instructions executable on the processor, and the program or instructions, when executed by the processor, implement the steps of the image processing method according to any one of claims 1-8. 18.A readable storage medium, wherein the readable storage medium stores a program or instructions, and the program or instructions, when executed by a processor, implement the steps of the image processing method according to any one of claims 1-8.

Images