Image processing method and device, electronic equipment and medium

Abstract

The invention discloses an image processing method and device, electronic equipment and a medium, and belongs to the technical field of image processing. The image processing method provided by the embodiment of the invention comprises the following steps: a first image signal processor performs abnormal frame detection processing on an image frame transmitted by an image sensor to obtain a detection result; when the detection result indicates that the image frame is an abnormal image frame, the first image signal processor carries out restoration processing on the abnormal image frame to obtain a restored image frame; the restored image frame is transmitted to a second image signal processor.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and specifically relates to an image processing method, device, electronic equipment and medium. Background Art

[0002] When transmitting image data between the image sensor and the external pre-processing image signal processor (ISP) chip (hereinafter referred to as the external ISP chip), the image transmission system based on the Mobile Industry Processor Interface (MIPI) may be subject to abnormal interference such as electromagnetic noise, signal crosstalk, and physical connection defects, resulting in data corruption. Damaged image data can cause system operation freezes and system crashes, making it difficult to ensure reliable transmission of image data. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide an image processing method, device, electronic device and medium, which can ensure the reliable transmission of abnormal image frames from the image sensor to the image signal processing module in the application processor, and effectively avoid system freezes or system crashes.

[0004] In a first aspect, an embodiment of the present application provides an image processing method, comprising:

[0005] The first image signal processor performs abnormal frame detection processing on the image frame transmitted by the image sensor to obtain a detection result;

[0006] The first image signal processor performs repair processing on the abnormal image frame to obtain a repaired image frame when the detection result indicates that the image frame is an abnormal image frame;

[0007] The restored image frame is transmitted to a second image signal processor.

[0008] In a second aspect, an image processing device is provided, including a first image signal processor, the device comprising:

[0009] a processing module, configured to perform abnormal frame detection processing on the image frame transmitted by the image sensor to obtain a detection result; and, if the detection result indicates that the image frame is an abnormal image frame, perform repair processing on the abnormal image frame to obtain a repaired image frame;

[0010] The sending module is used to transmit the restored image frame to the second image signal processor.

[0011] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0012] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0013] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0014] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the first aspect.

[0015] In an embodiment of the present application, the first image signal processor performs abnormal frame detection processing on the image frame transmitted by the image sensor to obtain a detection result; when the detection result indicates that the image frame is an abnormal image frame, the first image signal processor performs repair processing on the abnormal image frame to obtain a repaired image frame; and transmits the repaired image frame to the second image signal processor. As a result, damaged image frames or incomplete image frames in the image transmission process can be identified in time, and the identified abnormal image frames can be repaired to ensure the integrity of the abnormal image frames, thereby ensuring the reliability of the transmission of the abnormal image frames from the image sensor to the image signal processing module in the application processor, and effectively avoiding system freezes or system crashes. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a flowchart of an image processing method according to some embodiments of the present application;

[0017] Figure 2 is a module schematic diagram of an image processing device in some embodiments of the present application;

[0018] Figure 3 is a structural block diagram of an electronic device in some embodiments of the present application;

[0019] Figure 4 It is a structural block diagram of an electronic device of some embodiments of the present application. DETAILED DESCRIPTION

[0020] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0021] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0022] In order to enable those skilled in the art to better understand the embodiments of the present application, first, the following description is given.

[0023] It should be noted that the image processing method provided in the embodiments of the present application can be executed by electronic devices such as mobile phones, tablet computers, laptop computers, PDAs, and vehicle-mounted electronic devices. In some embodiments of the present application, the image processing method provided in the embodiments of the present application is described by taking the electronic device as the execution subject to execute the image processing method as an example.

[0024] The image processing method provided in the embodiment of the present application can be applied to scenarios in which a camera shoots images, wherein one specific application scenario is a photo preview scenario, and another specific application scenario is a video recording scenario.

[0025] The image processing method provided in the embodiment of the present application is described in detail below through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0026] like Figure 1 As shown, an embodiment of the present application provides an image processing method, the method comprising:

[0027] Step 101: a first image signal processor performs abnormal frame detection processing on image frames transmitted by an image sensor to obtain a detection result.

[0028] Optionally, the first image signal processor has an image preprocessing function, the first image signal processor is connected to the image sensor and the second image signal processor respectively, and the second image processor is an image signal processing module in the application processor.

[0029] In an embodiment of the present application, the first image signal processor is the external image signal processing (ISP) chip. The image preprocessing function includes but is not limited to initial denoising, color correction, and basic enhancement processing. Optionally, the abnormal image frame includes at least one of the following: an image frame with damaged pixel rows and an image frame with missing pixel rows.

[0030] In this step, the first image signal processor performs abnormal frame detection processing on the image frames transmitted by the image sensor in real time, so as to determine whether there are abnormal image frames in the image frames transmitted by the image sensor.

[0031] For example, in a photo preview scenario, the image sensor captures a preview image frame and transmits it to the first image signal processor. If the first image signal processor detects that at least one pixel row in the preview image frame is damaged, the preview image frame is determined to be an abnormal image frame.

[0032] For another example, in a video recording scenario, the image sensor transmits the image frame of the recorded video to the first image signal processor in real time. The first image signal processor detects that at least one pixel row of the image frame is missing, and determines that the image frame is an abnormal image frame.

[0033] Step 102: When the detection result indicates that the image frame is an abnormal image frame, the first image signal processor performs repair processing on the abnormal image frame to obtain a repaired image frame.

[0034] Optionally, when the detection result indicates that the image frame is an abnormal image frame, the first image signal processor does not perform image preprocessing on the abnormal image frame, and performs repair processing on the abnormal image frame to obtain a repaired image frame.

[0035] Optionally, the above-mentioned repair process is used to ensure the integrity of the abnormal image frame.

[0036] Optionally, when the detection result indicates that the image frame is not an abnormal image frame, an image preprocessing function is performed on the image frame.

[0037] In this step, no image preprocessing is performed on the abnormal image frame to prevent the image content of the abnormal image frame from affecting the image quality.

[0038] For example, in a photo preview scenario, if the detection result indicates that at least one pixel row in the preview image frame is damaged, the damaged pixel row of the preview image frame is repaired to complete the image content of the damaged pixel row to obtain a repaired image frame.

[0039] For another example, in a video recording scenario, if the detection result indicates that at least one pixel row of an image frame of the recorded video is missing, the missing pixel row in the image frame is padded to obtain a repaired image frame.

[0040] Step 103: Transmit the restored image frame to a second image signal processor.

[0041] For example, in a photo preview scenario, the restored preview image frame is transmitted to the second image signal processor; for another example, in a video recording scenario, the restored video image frame is transmitted to the second image signal processor.

[0042] Here, the repaired image frame is transmitted to the second image signal processor to ensure reliable transmission of the image data, and the second image signal processor adopts a corresponding exception handling strategy after receiving the repaired image frame.

[0043] In an embodiment of the present application, the first image signal processor performs abnormal frame detection processing on the image frame transmitted by the image sensor to obtain a detection result; when the detection result indicates that the image frame is an abnormal image frame, the first image signal processor performs repair processing on the abnormal image frame to obtain a repaired image frame; and transmits the repaired image frame to the second image signal processor. As a result, damaged image frames or incomplete image frames in the image transmission process can be identified in time, and the identified abnormal image frames can be repaired to ensure the integrity of the abnormal image frames, thereby ensuring the reliability of the transmission of the abnormal image frames from the image sensor to the image signal processing module in the application processor, and effectively avoiding system freezes or system crashes.

[0044] Optionally, the first image signal processor performs abnormal frame detection processing on the image frame transmitted by the image sensor to obtain a detection result, including:

[0045] Acquire standard image parameters corresponding to sensor parameters of the image sensor, the standard image parameters including at least one of the following: a standard number of lines and a standard line interval of an image frame, the sensor parameters including at least one of the following: a configuration parameter of the sensor and an operation mode parameter of the sensor;

[0046] The detection result of the image frame is determined according to the relationship between the parameter difference and the parameter difference threshold, wherein the parameter difference is the difference between the parameter value of the image parameter of the image frame output by the image sensor and the parameter value of the standard image parameter.

[0047] Optionally, the configuration parameters of the sensor include but are not limited to pixel size, resolution, frame rate, etc. The operation mode parameters of the sensor include a shooting mode, for example, the shooting mode may be a general shooting mode, a high dynamic range shooting mode, or a preview mode.

[0048] In some embodiments of the present application, standard image parameters are configured based on sensor parameters of the image sensor, and different sensor parameters correspond to different standard image parameters. For example, different sensor parameters correspond to different numbers of rows and row intervals.

[0049] As an implementation manner, the standard image parameters corresponding to the sensor parameters of the above-mentioned image sensor are acquired based on a technical manual of the image sensor or a technical specification document provided by a sensor manufacturer.

[0050] For example, in the photo preview scene, the sensor parameters of the image sensor correspond to the standard number of rows of the image frame A1, and the sensor parameters of the image sensor correspond to the interval of the mark rows of the image frame B1;

[0051] In the video recording scenario, the sensor parameters of the image sensor correspond to a standard number of A2 rows of the image frame, and the sensor parameters of the image sensor correspond to a marker row interval of the image frame of B2, wherein A1 is different from A2, and B1 is different from B2.

[0052] Optionally, as an implementation method, the detection result of the image frame is determined based on the relationship between the parameter difference and the parameter difference threshold, including: if the parameter difference is greater than the parameter difference threshold, the image frame is determined to be an abnormal image frame, otherwise, the image frame is determined to be a normal image frame.

[0053] In some embodiments of the present application, the image parameters of the image frame transmitted by the image sensor are compared with the standard image parameters; if the difference between the value of the image parameter of the image frame transmitted by the image sensor and the standard image parameter is greater than a preset threshold, the image frame is determined to be an abnormal image frame. For example, if the number of rows of the image frame transmitted by the image sensor is 260 rows, the pixel row interval is 15 pixels, the standard number of rows is 300 rows, the standard pixel row interval is 30 pixels, the row number difference threshold is 10, the pixel row interval threshold is 10, the number of rows of the image frame and the standard number of rows The difference is 40 rows, the row number difference is greater than the row number difference threshold, the difference between the pixel row interval of the image frame and the standard pixel row interval is 15 pixels, and the pixel row interval difference is greater than the pixel row interval threshold, therefore, the image frame is determined to be an abnormal image frame.

[0054] When the difference between the value of the image parameter of the image frame transmitted by the image sensor and the value of the standard image parameter is greater than the parameter difference threshold, it means that the deviation between the two exceeds the preset deviation, and the image frame is determined to be an abnormal image frame. The parameter difference threshold can be set based on the stable operating performance of the image sensor and the minimum allowable error.

[0055] In some embodiments of the present application, a real-time monitoring algorithm is used to monitor whether each image frame transmitted by the image sensor is an abnormal image frame, and a set of dedicated circuits capable of executing the real-time monitoring algorithm is designed according to the specifications of the image sensor and the selected configuration to ensure that it can perform real-time analysis on the arriving image frames, so that abnormal frames can be quickly identified when they appear, thereby ensuring the integrity and reliability of image transmission during image transmission based on the MIPI interface.

[0056] Optionally, the repairing the abnormal image frame to obtain a repaired image frame includes:

[0057] Identifying abnormal pixel rows in the abnormal image frame, wherein the abnormal pixel rows are missing or damaged rows in the abnormal image frame;

[0058] According to the reference image data, repairing the image data of the abnormal pixel row in the abnormal image frame to obtain a repaired image frame;

[0059] Wherein, the reference image data includes: image data of the abnormal pixel row of the reference image frame; or image data of the reference pixel row of the abnormal image frame, the frame interval between the reference image frame and the abnormal image frame is less than a frame interval threshold, and the line interval between the reference pixel row and the abnormal pixel row is less than a line interval threshold.

[0060] In some embodiments of the present application, the above-mentioned image data is the image content of the image frame.

[0061] As an implementation method, the above-mentioned reference image frame is an adjacent image frame of the abnormal image frame. For example, the reference image frame is the previous image frame or the next image frame of the abnormal image frame. The image data of the abnormal image frame is padded based on the image data of the adjacent image frame to ensure the continuity of the image output, thereby effectively ensuring the reliability of image transmission.

[0062] As an implementation manner, the reference pixel row is an adjacent pixel row of the abnormal pixel row.

[0063] For example, when the image data of the abnormal pixel row of the reference image frame is unavailable, a difference algorithm is used to generate the image data of the abnormal pixel row based on the image data of adjacent pixel rows of the abnormal pixel row of the abnormal image frame.

[0064] Exemplarily, in a photo preview scenario, assuming that rows 10-15 of the abnormal preview image frame are abnormal pixel rows, the image data of rows 10-15 may be generated based on the image data of rows 5-9 of the abnormal preview image frame.

[0065] Exemplarily, in a video recording scenario, rows 1-5 of the third image frame of the recorded video are abnormal pixel rows, and image data of rows 1-5 of the third image frame can be generated based on image data of rows 1-5 of the second image frame of the recorded video.

[0066] Optionally, performing repair processing on the image data of the abnormal pixel row in the abnormal image frame includes:

[0067] When the number of the abnormal pixel rows is less than the pixel row threshold, the image data of the abnormal pixel rows in the abnormal image frame are repaired according to the reference image data.

[0068] When the number of abnormal pixel rows is greater than or equal to a pixel row threshold, at least one of the image sensor, the first image signal processor, and the second image signal processor is restarted or reset.

[0069] When the number of abnormal pixel rows is less than the pixel row threshold, it indicates that the abnormal image frame is image damage caused by abnormal interference, rather than image damage caused by device abnormality. Therefore, the image data of the target row in the abnormal image frame can be repaired to ensure reliable transmission of image data.

[0070] In an embodiment of the present application, abnormal pixel rows in the abnormal image frame are supplemented according to reference image data to ensure the continuity of image output, thereby effectively ensuring the reliability of image transmission.

[0071] Optionally, the repairing the abnormal image frame to obtain a repaired image frame includes:

[0072] Among the cached image frames, an image frame whose difference between a timestamp and a timestamp of an abnormal image frame is less than a timestamp threshold is selected as a repaired image frame.

[0073] Optionally, the above timestamp can be understood as the start scanning time of the image frame.

[0074] As an implementation manner, among the cached image frames, an image frame whose timestamp has the smallest difference with the timestamp of the abnormal image frame is selected as the repaired image frame.

[0075] Optionally, in the embodiment of the present application, the first image signal processor caches a preset number of image frames and continuously updates the cached image frames, wherein the cached image frames do not include abnormal image frames, that is, the cached image frames are valid image frames. For example, the first image signal processor caches three image frames, namely, the first to third image frames, and after receiving the fourth image frame, the cached image frames are updated to the second to fourth image frames.

[0076] Exemplarily, the first image signal processor caches three image frames, namely, image frame S1, image frame S2, and image frame S3, and the timestamps of the three image frames are T1, T2, and T3, respectively, wherein the time corresponding to T1 is earlier than the time corresponding to T2, and the time corresponding to T2 is earlier than the time corresponding to T3. After receiving image frame S4, the first image signal processor detects that image frame S4 ​​is an abnormal image frame, then compares the timestamp T4 of image frame S4 ​​with the above T1, T2, and T3, respectively, and determines that T3 is closest to T4, then uses image frame S4 ​​as a repaired image frame.

[0077] In an embodiment of the present application, in order to avoid image interruption caused by frame loss, the first image signal processor continuously caches a certain number of valid image frames internally. When an abnormal image frame is detected, a frame whose timestamp difference is less than a timestamp threshold is selected from the cached image frames as a repaired image frame to ensure the continuity and stability of the image output.

[0078] Optionally, the method of the embodiment of the present application further includes:

[0079] The first image signal processor obtains indication information of the image frame transmitted by the image sensor according to the reference information, wherein the indication information is used to indicate whether the image frame is an abnormal image frame;

[0080] sending the indication information to the second image signal processor;

[0081] The reference information includes at least one of the following: a control data bit of the image frame, image content indication information of the image frame, and an error flag bit.

[0082] In some embodiments of the present application, the data at a specific position of the content transmitted from the first image signal processor to the second image signal processor, for example, the first segment of data or the last segment of data of the transmitted content, is used to determine the control data bit, and the data corresponding to the control data bit does not include the image content of the image frame. For example, the control data bit corresponding to the normal image frame is data 1, and the control data bit corresponding to the abnormal image frame is data 2.

[0083] In some embodiments of the present application, the above-mentioned image content indication information may refer to data information at a specific position in the image content corresponding to the image frame. For example, if the data information at a specific position in the image content corresponding to the image frame is all 0, the image frame is determined to be an abnormal image frame.

[0084] In some embodiments of the present application, the error flag may be an error code in a log or image metadata, and the error code is used to facilitate the application processor to correctly distinguish between normal image frames and abnormal image frames. For example, if the error flag is 1, it indicates that the image frame is a normal image frame, and if the error flag is 0, it indicates that the image frame is an abnormal image frame.

[0085] In addition, the above logs and image metadata can also pass error codes containing detailed exception reasons and conditions to the application processor, and can also provide relevant frame numbers and timestamps to facilitate logging and subsequent problem tracking by the application processor.

[0086] In an embodiment of the present application, when an abnormal image frame is detected, indication information for indicating whether the image frame is an abnormal image frame is obtained through reference information and transmitted to the second image signal processor so that the second image signal processor can identify the abnormal image frame. In addition, the first image signal processor filters out the abnormal image frame from the normal processing flow, that is, does not perform image preprocessing on the abnormal image frame to avoid a negative impact on image quality.

[0087] Optionally, the method of the embodiment of the present application further includes:

[0088] The first image signal processor sends an interrupt signal to the second image signal processor, where the interrupt signal is used to trigger a recognition process of abnormal image frames.

[0089] The interrupt signal may be a hardware interrupt request signal (Interrupt Request, IRQ), and the interrupt type corresponding to the interrupt request signal includes but is not limited to: abnormal data packet, missing data line, data packet timeout, etc.

[0090] Exemplarily, after receiving the interrupt signal, the second image signal processor reads the above indication information through the corresponding driver to determine whether the image frame is an abnormal image frame. The second image signal processor will perform different response measures according to the type and severity of the abnormality. This may include: ignoring abnormal frames with smaller defects, restoring sensor settings, or restarting hardware and software systems. In addition, the second image signal processor can record all abnormal events in the system log for subsequent analysis to improve the robustness of the system and assist in future error-proofing design.

[0091] In the embodiment of the present application, when the first image signal processor detects an abnormal image frame, it sends an interrupt signal so that the second signal processor can identify the abnormal image frame based on the interrupt signal and then adopt a corresponding processing strategy for the abnormal image frame.

[0092] The above solution can significantly improve the reliability and stability of the external ISP chip in practical applications, and reduce the impact of abnormal frames on the final image quality through the internal exception handling mechanism. These measures not only ensure the integrity of image data, but also provide flexibility and adaptability in operations under abnormal conditions.

[0093] Optionally, the method of the embodiment of the present application further includes:

[0094] When the number of abnormal image frames in the image frames transmitted by the image sensor is greater than a frame number threshold, the first image signal processor performs at least one of the following:

[0095] Performing a reset process, wherein the reset process includes at least one of the following: reinitializing an interface of the image sensor, resetting a configuration of the image sensor;

[0096] Output warning information.

[0097] The above configuration of resetting the image sensor may be understood as performing a reset process on the image sensor.

[0098] In some embodiments of the present application, the configuration of the image sensor is used to configure the image sensor to collect image data, and the configuration of the image sensor includes but is not limited to resolution, frame rate, image format, etc.

[0099] By adopting the above reset processing strategy to ensure that the image sensor is restored to a normal working state, of course, the reset processing can also include other operations that can restore the image sensor to a normal working state, which are not specifically limited here. By outputting alarm information, the external control system is informed to take additional intervention measures, such as switching backup sensors or performing system-level error troubleshooting and repair.

[0100] Optionally, the first image signal processor is connected to the image sensor and the second image signal processor respectively, and the first image signal processor has an image preprocessing function;

[0101] The second image signal processor is an image signal processing module in an application processor (AP).

[0102] The first image signal processor mentioned above can also be described as an external ISP.

[0103] The image preprocessing function of the first image signal processor has been explained in the above description and will not be repeated here.

[0104] In an embodiment of the present application, a first image signal processor having an image preprocessing function is arranged between the image sensor and the second image signal processor. In addition to the image preprocessing function, the first image signal processor can also perform abnormal frame detection and abnormal frame repair processing on the image frames transmitted by the image sensor, thereby ensuring the reliable transmission of abnormal image frames from the image sensor to the image signal processing module in the application processor, and effectively avoiding system freezes or system crashes.

[0105] In the embodiment of the present application, a comprehensive and reliable abnormal information transmission and processing mechanism is established between the external ISP and the AP. This mechanism can identify multiple abnormal types and provide corresponding abnormal processing strategies for each type. Through this solution, the system can efficiently handle abnormal situations, reduce the impact of abnormal events on the end-user experience, and thereby improve the stability and reliability of the entire imaging system.

[0106] It should be noted that the solution of this application has wide applicability and scalability, and is not only applicable to the current mobile phone imaging field, but also applicable to the following possible application scenarios:

[0107] Medical imaging systems: In high-precision imaging systems such as X-ray machines and CT scans, image quality directly affects the accuracy of diagnostic results. It is crucial to ensure the integrity and reliability of imaging data using the solution of this application.

[0108] In-vehicle camera system: In autonomous driving and advanced driving assistance systems (ADAS), in-vehicle cameras provide key visual information. Rapid detection and correction of abnormal frames can improve road safety and reduce risks caused by image delays or errors.

[0109] Security monitoring: Security camera systems require 24 / 7 uninterrupted operation and extremely high data reliability. Applying this application can help the system detect and handle abnormal images in a timely manner and ensure the continuity of monitoring data.

[0110] Motion capture and analysis: In motion capture for sports games or animation production, distorted images will affect the accuracy of motion data analysis. Identifying and processing abnormal frames in advance can provide more stable and reliable data output.

[0111] Aerospace remote sensing: In satellite or drone-borne imaging systems, high-quality image data is essential for the collection and analysis of geographic information. This application ensures that the quality of image data is maintained as much as possible even in bad weather or signal interference conditions.

[0112] Industrial visual inspection system: In automated production lines, image detection is used to monitor product quality. Detection and processing of abnormal frames can correct image data errors in real time to ensure consistent production quality.

[0113] The image processing method provided in the embodiment of the present application can be executed by an image processing device. In the embodiment of the present application, an image processing device executing the image processing method is taken as an example to illustrate the image processing device provided in the embodiment of the present application.

[0114] like Figure 2 As shown, the embodiment of the present application further provides an image processing device 200, including a first image signal processor, the device comprising:

[0115] The processing module 201 is used to perform abnormal frame detection processing on the image frame transmitted by the image sensor to obtain a detection result; if the detection result indicates that the image frame is an abnormal image frame, perform repair processing on the abnormal image frame to obtain a repaired image frame;

[0116] The sending module 202 is used to transmit the restored image frame to the second image signal processor.

[0117] Optionally, the processing module is specifically used for:

[0118] Acquire standard image parameters corresponding to sensor parameters of the image sensor, the standard image parameters including at least one of the following: a standard number of lines and a standard line interval of an image frame, the sensor parameters including at least one of the following: a configuration parameter of the sensor and an operation mode parameter of the sensor;

[0119] The detection result of the image frame is determined according to the relationship between the parameter difference and the parameter difference threshold, wherein the parameter difference is the difference between the parameter value of the image parameter of the image frame output by the image sensor and the parameter value of the standard image parameter.

[0120] Optionally, the processing module is specifically used for:

[0121] Identifying abnormal pixel rows in the abnormal image frame, wherein the abnormal pixel rows are missing or damaged rows in the abnormal image frame;

[0122] According to the reference image data, repairing the image data of the abnormal pixel row in the abnormal image frame to obtain a repaired image frame;

[0123] Wherein, the reference image data includes: image data of the abnormal pixel row of the reference image frame; or image data of the reference pixel row of the abnormal image frame, the frame interval between the reference image frame and the abnormal image frame is less than a frame interval threshold, and the line interval between the reference pixel row and the abnormal pixel row is less than a line interval threshold.

[0124] Optionally, the processing module is specifically used for:

[0125] Among the cached image frames, an image frame whose difference between a timestamp and a timestamp of an abnormal image frame is less than a timestamp threshold is selected as a repaired image frame.

[0126] Optionally, the processing module is further used for:

[0127] Obtaining indication information of an image frame transmitted by an image sensor according to the reference information, wherein the indication information is used to indicate whether the image frame is an abnormal image frame;

[0128] sending the indication information to the second image signal processor;

[0129] The reference information includes at least one of the following: a control data bit of the image frame, image content indication information of the image frame, and an error flag bit.

[0130] Optionally, the sending module is further used for:

[0131] An interrupt signal is sent to the second image signal processor, where the interrupt signal is used to trigger a recognition process of abnormal image frames.

[0132] Optionally, the processing module is further used for:

[0133] When the number of abnormal image frames in the image frames transmitted by the image sensor is greater than a frame number threshold, at least one of the following is performed:

[0134] Performing a reset process, wherein the reset process includes at least one of the following: reinitializing an interface of the image sensor, resetting a configuration of the image sensor;

[0135] Output warning information.

[0136] Optionally, the first image signal processor is connected to the image sensor and the second image signal processor respectively, and the first image signal processor has an image preprocessing function;

[0137] The second image signal processor is an image signal processing module in the application processor.

[0138] In the embodiment of the present application, the first image signal processor performs abnormal frame detection processing on the image frame transmitted by the image sensor to obtain a detection result; when the detection result indicates that the image frame is an abnormal image frame, the first image signal processor performs repair processing on the abnormal image frame to obtain a repaired image frame; and transmits the repaired image frame to the second image signal processor. As a result, damaged image frames or incomplete image frames in the image transmission process can be identified in time, and the identified abnormal image frames can be repaired to ensure the integrity of the abnormal image frames, thereby ensuring the reliability of the transmission of the abnormal image frames from the image sensor to the image signal processing module in the application processor, and effectively avoiding system freezes or system crashes.

[0139] The image processing device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0140] The image processing device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0141] The image processing device provided in the embodiment of the present application can achieve Figure 1 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0142] Alternatively, if Figure 3As shown, an embodiment of the present application further provides an electronic device 300, including a processor 301 and a memory 302, wherein the memory 302 stores programs or instructions that can be executed on the processor 301, and when the program or instructions are executed by the processor 301, the various steps of the above-mentioned image processing method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0143] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0144] Figure 4 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0145] The electronic device 400 includes but is not limited to components such as a radio frequency unit 401 , a network module 402 , an audio output unit 403 , an input unit 404 , a sensor 405 , a display unit 406 , a user input unit 407 , an interface unit 408 , a memory 409 , and a processor 410 .

[0146] Those skilled in the art will appreciate that the electronic device 400 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 410 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Figure 4 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0147] Among them, the processor 410 is used to perform abnormal frame detection processing on the image frame transmitted by the image sensor to obtain a detection result; when the detection result indicates that the image frame is an abnormal image frame, the abnormal image frame is repaired to obtain a repaired image frame; the radio frequency unit 401 is used to transmit the repaired image frame to the second image signal processor.

[0148] Optionally, the processor 410 is specifically configured to:

[0149] Acquire standard image parameters corresponding to sensor parameters of the image sensor, the standard image parameters including at least one of the following: a standard number of lines and a standard line interval of an image frame, the sensor parameters including at least one of the following: a configuration parameter of the sensor and an operation mode parameter of the sensor;

[0150] The detection result of the image frame is determined according to the relationship between the parameter difference and the parameter difference threshold, wherein the parameter difference is the difference between the parameter value of the image parameter of the image frame output by the image sensor and the parameter value of the standard image parameter.

[0151] Optionally, the processor 410 is specifically configured to:

[0152] Identifying abnormal pixel rows in the abnormal image frame, wherein the abnormal pixel rows are missing or damaged rows in the abnormal image frame;

[0153] According to the reference image data, repairing the image data of the abnormal pixel row in the abnormal image frame to obtain a repaired image frame;

[0154] Wherein, the reference image data includes: image data of the abnormal pixel row of the reference image frame; or image data of the reference pixel row of the abnormal image frame, the frame interval between the reference image frame and the abnormal image frame is less than a frame interval threshold, and the line interval between the reference pixel row and the abnormal pixel row is less than a line interval threshold.

[0155] Optionally, the processor 410 is specifically configured to:

[0156] Among the cached image frames, an image frame whose difference between a timestamp and a timestamp of an abnormal image frame is less than a timestamp threshold is selected as a repaired image frame.

[0157] Optionally, the processor 410 is further configured to:

[0158] The first image signal processor obtains indication information of the image frame transmitted by the image sensor according to the reference information, wherein the indication information is used to indicate whether the image frame is an abnormal image frame;

[0159] sending the indication information to the second image signal processor;

[0160] The reference information includes at least one of the following: a control data bit of the image frame, image content indication information of the image frame, and an error flag bit.

[0161] Optionally, the radio frequency unit 401 is further configured to:

[0162] The first image signal processor sends an interrupt signal to the second image signal processor, where the interrupt signal is used to trigger a recognition process of abnormal image frames.

[0163] Optionally, the processor 410 is further configured to:

[0164] When the number of abnormal image frames in the image frames transmitted by the image sensor is greater than a frame number threshold, the first image signal processor performs at least one of the following:

[0165] Performing a reset process, wherein the reset process includes at least one of the following: reinitializing an interface of the image sensor, resetting a configuration of the image sensor;

[0166] Output warning information.

[0167] Optionally, the first image signal processor is connected to the image sensor and the second image signal processor respectively, and the first image signal processor has an image preprocessing function;

[0168] The second image signal processor is an image signal processing module in the application processor.

[0169] In the embodiment of the present application, the first image signal processor performs abnormal frame detection processing on the image frame transmitted by the image sensor to obtain a detection result; when the detection result indicates that the image frame is an abnormal image frame, the first image signal processor performs repair processing on the abnormal image frame to obtain a repaired image frame; and transmits the repaired image frame to the second image signal processor. As a result, damaged image frames or incomplete image frames in the image transmission process can be identified in time, and the identified abnormal image frames can be repaired to ensure the integrity of the abnormal image frames, thereby ensuring the reliability of the transmission of the abnormal image frames from the image sensor to the image signal processing module in the application processor, and effectively avoiding system freezes or system crashes.

[0170] It should be understood that in the embodiment of the present application, the input unit 404 may include a graphics processor (Graphics Processing Unit, GPU) 4041 and a microphone 4042, and the graphics processor 4041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 406 may include a display panel 4061, and the display panel 4061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 407 includes a touch panel 4071 and at least one of other input devices 4072. The touch panel 4071 is also called a touch screen. The touch panel 4071 may include two parts: a touch detection device and a touch controller. Other input devices 4072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0171] The memory 409 can be used to store software programs and various data. The memory 409 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 409 may include a volatile memory or a non-volatile memory, or the memory 409 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 409 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0172] The processor 410 may include one or more processing units; optionally, the processor 410 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 410.

[0173] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned image processing method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0174] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0175] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned image processing method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0176] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0177] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned image processing method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0178] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0179] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0180] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. An image processing method, characterized in that: include: The first image signal processor performs abnormal frame detection processing on the image frame transmitted by the image sensor to obtain a detection result; The first image signal processor performs repair processing on the abnormal image frame to obtain a repaired image frame when the detection result indicates that the image frame is an abnormal image frame; The restored image frame is transmitted to a second image signal processor.

2. The method according to claim 1, characterized in that The first image signal processor performs abnormal frame detection processing on the image frame transmitted by the image sensor to obtain a detection result, including: Acquire standard image parameters corresponding to sensor parameters of the image sensor, wherein the standard image parameters include at least one of the following items of the image frame: a standard number of lines and a standard line interval, and the sensor parameters include at least one of the following items: a configuration parameter of the sensor and an operation mode parameter of the sensor; The detection result of the image frame is determined according to the relationship between the parameter difference and the parameter difference threshold, wherein the parameter difference is the difference between the parameter value of the image parameter of the image frame output by the image sensor and the parameter value of the standard image parameter.

3. The method according to claim 1, characterized in that: The repairing process of the abnormal image frame to obtain the repaired image frame includes: Identifying abnormal pixel rows in the abnormal image frame, wherein the abnormal pixel rows are missing or damaged rows in the abnormal image frame; According to the reference image data, repairing the image data of the abnormal pixel row in the abnormal image frame to obtain a repaired image frame; Wherein, the reference image data includes: image data of the abnormal pixel row of the reference image frame; or image data of the reference pixel row of the abnormal image frame, the frame interval between the reference image frame and the abnormal image frame is less than a frame interval threshold, and the line interval between the reference pixel row and the abnormal pixel row is less than a line interval threshold.

4. The method according to claim 1, characterized in that: The repairing process of the abnormal image frame to obtain the repaired image frame includes: Among the cached image frames, an image frame whose difference between a timestamp and a timestamp of an abnormal image frame is less than a timestamp threshold is selected as a repaired image frame.

5. The method according to claim 1, characterized in that The method further comprises: The first image signal processor obtains indication information of the image frame transmitted by the image sensor according to the reference information, wherein the indication information is used to indicate whether the image frame is an abnormal image frame; sending the indication information to the second image signal processor; The reference information includes at least one of the following: a control data bit of the image frame, image content indication information of the image frame, and an error flag bit.

6. The method according to claim 1, characterized in that The method further comprises: The first image signal processor sends an interrupt signal to the second image signal processor, where the interrupt signal is used to trigger a recognition process of abnormal image frames.

7. The method according to claim 1, characterized in that The method further comprises: When the number of abnormal image frames in the image frames transmitted by the image sensor is greater than a frame number threshold, the first image signal processor performs at least one of the following: Performing a reset process, wherein the reset process includes at least one of the following: reinitializing an interface of the image sensor, resetting a configuration of the image sensor; Output warning information.

8. The method according to claim 1, characterized in that The first image signal processor is connected to the image sensor and the second image signal processor respectively, and the first image signal processor has an image preprocessing function; The second image signal processor is an image signal processing module in the application processor.

9. An image processing device, comprising a first image signal processor, characterized in that: The device comprises: a processing module, configured to perform abnormal frame detection processing on the image frame transmitted by the image sensor to obtain a detection result; and, if the detection result indicates that the image frame is an abnormal image frame, perform repair processing on the abnormal image frame to obtain a repaired image frame; The sending module is used to transmit the restored image frame to the second image signal processor.

10. The device according to claim 9, characterized in that The processing module is specifically used for: Acquire standard image parameters corresponding to sensor parameters of the image sensor, the standard image parameters including at least one of the following: a standard number of lines and a standard line interval of an image frame, the sensor parameters including at least one of the following: a configuration parameter of the sensor and an operation mode parameter of the sensor; The detection result of the image frame is determined according to the relationship between the parameter difference and the parameter difference threshold, wherein the parameter difference is the difference between the parameter value of the image parameter of the image frame output by the image sensor and the parameter value of the standard image parameter.

11. The device according to claim 9, characterized in that The processing module is specifically used for: Identifying abnormal pixel rows in the abnormal image frame, wherein the abnormal pixel rows are missing or damaged rows in the abnormal image frame; According to the reference image data, repairing the image data of the abnormal pixel row in the abnormal image frame to obtain a repaired image frame; Wherein, the reference image data includes: image data of the abnormal pixel row of the reference image frame; or image data of the reference pixel row of the abnormal image frame, the frame interval between the reference image frame and the abnormal image frame is less than a frame interval threshold, and the line interval between the reference pixel row and the abnormal pixel row is less than a line interval threshold.

12. The device according to claim 9, characterized in that The processing module is specifically used for: Among the cached image frames, an image frame whose difference between a timestamp and a timestamp of an abnormal image frame is less than a timestamp threshold is selected as a repaired image frame.

13. The device according to claim 9, characterized in that The processing module is also used for: Obtaining indication information of an image frame transmitted by an image sensor according to the reference information, wherein the indication information is used to indicate whether the image frame is an abnormal image frame; sending the indication information to the second image signal processor; The reference information includes at least one of the following: a control data bit of the image frame, image content indication information of the image frame, and an error flag bit.

14. The device according to claim 9, characterized in that The sending module is also used for: An interrupt signal is sent to the second image signal processor, where the interrupt signal is used to trigger a recognition process of abnormal image frames.

15. The device according to claim 9, characterized in that The processing module is also used for: When the number of abnormal image frames in the image frames transmitted by the image sensor is greater than a frame number threshold, at least one of the following is performed: Performing a reset process, wherein the reset process includes at least one of the following: reinitializing an interface of the image sensor, resetting a configuration of the image sensor; Output warning information.

16. The device according to claim 9, characterized in that The first image signal processor is connected to the image sensor and the second image signal processor respectively, and the first image signal processor has an image preprocessing function; The second image signal processor is an image signal processing module in the application processor.

17. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the image processing method according to any one of claims 1 to 8 are implemented.

18. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the image processing method according to any one of claims 1 to 8 are implemented.

Images