Function safety-based image frame processing method and apparatus, storage medium, and device

Abstract

Disclosed are a function safety-based image frame processing method and apparatus, a storage medium, and a device. The function safety-based image frame processing method comprises: during image format conversion of a first image frame from a first image type to a second image type, determining error information of a hardware module in which a function safety error has occurred; on the basis of the error information, determining a reliability assessment result of a second image frame converted to the second image type; determining a data attribute of the second image frame on the basis of the reliability assessment result of the second image frame; and performing post-processing on the second image frame according to a post-processing approach adapted to the data attribute. The embodiments of the present disclosure can avoid adverse impact caused by function safety errors in hardware modules in an image processing pipeline.

Description

Image frame processing method, device, storage medium and equipment based on functional safety

[0001] This disclosure claims priority to Chinese patent application number 202410175803.4, filed on February 7, 2024, entitled “Image frame processing method, device, medium and equipment based on functional safety”, the entire contents of which are incorporated by reference into this disclosure. Technical Field

[0002] The present disclosure relates to image processing technology, and in particular to a method, apparatus, storage medium, and device for image frame processing based on functional safety. Background Art

[0003] In some scenarios, image frames need to be processed through an image processing pipeline, and the results of this pipeline are then used for subsequent processing. It should be noted that during this process, hardware modules within the pipeline may experience functional safety errors, resulting in adverse effects. Summary of the Invention

[0004] In order to solve the above technical problems, the present disclosure provides a functional safety-based image frame processing method, apparatus, storage medium and device to minimize the adverse effects caused by functional safety errors in hardware modules in the image processing pathway during the process of image frame processing by the image processing pathway.

[0005] According to one aspect of an embodiment of the present disclosure, a method for processing an image frame based on functional safety is provided, including:

[0006] During a process of converting an image format of a first image frame of a first image type into a second image type, determining error information of a hardware module in which a functional safety error occurs;

[0007] determining a reliability assessment result of a second image frame converted to the second image type based on the error information;

[0008] determining a data attribute of the second image frame based on a reliability evaluation result of the second image frame;

[0009] The second image frame is post-processed according to a post-processing method adapted to the data attributes.

[0010] According to another aspect of an embodiment of the present disclosure, there is provided an image frame processing device based on functional safety, comprising:

[0011] a first determining module, configured to determine error information of a hardware module having a functional safety error during a process of converting an image format of a first image frame of a first image type into a second image type;

[0012] a second determining module, configured to determine a reliability evaluation result of a second image frame converted to the second image type based on the error information determined by the first determining module;

[0013] a third determining module, configured to determine a data attribute of the second image frame based on the reliability evaluation result of the second image frame determined by the second determining module;

[0014] A post-processing module is used to post-process the second image frame according to a post-processing method that is compatible with the data attribute determined by the third determination module.

[0015] According to another aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and the computer program is used to execute the above-mentioned image frame processing method based on functional safety.

[0016] According to another aspect of the embodiments of the present disclosure, an electronic device is provided, the electronic device including:

[0017] processor;

[0018] a memory for storing instructions executable by the processor;

[0019] The processor is configured to read the executable instructions from the memory and execute the executable instructions to implement the above-mentioned image frame processing method based on functional safety.

[0020] According to another aspect of an embodiment of the present disclosure, a computer program product is provided. When instructions in the computer program product are executed by a processor, the above-mentioned image frame processing method based on functional safety is performed.

[0021] Based on the functional safety-based image frame processing methods, devices, storage media, electronic devices, and computer program products provided by the aforementioned embodiments of the present disclosure, during the process of processing a first image frame of a first image type through an image processing pathway to obtain a second image frame of a second image type, error information of a hardware module in the image processing pathway that has experienced a functional safety error can be determined. Based on the error information of the hardware module experiencing the functional safety error, a reliability assessment can be performed on the second image frame to obtain a reliability assessment result for the second image frame, thereby determining the data attributes of the second image frame. Based on the data attributes of the second image frame, the second image frame can be post-processed using an appropriate post-processing method. It should be noted that the data attributes of the second image frame are closely related to the error information of the hardware module experiencing the functional safety error. Accordingly, the post-processing method adopted for the second image frame is also closely related to the error information of the hardware module experiencing the functional safety error. That is, the embodiments of the present disclosure collect functional safety information of each hardware module in the image processing pathway as a whole and perform a decision analysis based on the overall picture to ensure the rationality of the post-processing method adopted for the second image frame. This helps to avoid the adverse effects caused by functional safety errors in the hardware modules in the image processing pathway. For example, the reliability of a hardware system circuit (eg, a system-on-chip) where the image processing path is located can be guaranteed. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] FIG1 is a diagram showing the working principle of an image processing pathway in some exemplary embodiments of the present disclosure;

[0023] FIG2 is a system architecture diagram in some exemplary embodiments of the present disclosure;

[0024] FIG3 is a flowchart of an image frame processing method based on functional safety provided by some exemplary embodiments of the present disclosure;

[0025] FIG4 is a flowchart of a method for determining a reliability evaluation result of a second image frame provided by some exemplary embodiments of the present disclosure;

[0026] FIG5 is a flowchart of a method for determining level weights corresponding to respective preset error levels provided by some exemplary embodiments of the present disclosure;

[0027] FIG6 is a flowchart of a method for determining a reliability evaluation result of a second image frame provided by some other exemplary embodiments of the present disclosure;

[0028] FIG7 is a flowchart of a method for determining data attributes of a second image frame provided by some exemplary embodiments of the present disclosure;

[0029] FIG8 is a flowchart of a method for updating error information provided by some exemplary embodiments of the present disclosure;

[0030] FIG9 is a schematic diagram illustrating a method for preventing adverse effects caused by functional safety errors in hardware modules in an image processing path according to some exemplary embodiments of the present disclosure;

[0031] FIG10 is a schematic structural diagram of an image frame processing device based on functional safety provided by some exemplary embodiments of the present disclosure;

[0032] FIG11 is a schematic structural diagram of a second determination module in some exemplary embodiments of the present disclosure;

[0033] FIG12 is a schematic structural diagram of a third determination module in some exemplary embodiments of the present disclosure;

[0034] FIG13 is a schematic structural diagram of an image frame processing device based on functional safety provided by other exemplary embodiments of the present disclosure;

[0035] FIG14 is a structural diagram of an electronic device provided by some exemplary embodiments of the present disclosure. DETAILED DESCRIPTION

[0036] To explain the present disclosure, example embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments. It should be understood that the present disclosure is not limited to the example embodiments.

[0037] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.

[0038] Application Overview

[0039] In some scenarios, it is necessary to process the image frame through an image processing pipeline, and further use the processing results of the image processing pipeline for subsequent processing.

[0040] In one example, as shown in FIG. 1 , the image processing pathway may include a first hardware module 110 , a second hardware module 120 , a third hardware module 130 , a fourth hardware module 140 , and a fifth hardware module 150 .

[0041] Optionally, the first hardware module 110 may include a Mobile Industry Processor Interface (MIPI). The first hardware module 110 may be configured to receive raw image frames captured by an image sensor and transmit the raw image frames to the second hardware module 120. The second hardware module 120 may transmit the raw image frames from the first hardware module 110 to the third hardware module 130. The raw image frames may also be referred to as raw data.

[0042] Optionally, the third hardware module 130 may include an image signal processor (ISP). The third hardware module 130 may perform predetermined processing on the raw image frames from the second hardware module 120 and output image frames of a specific image type (hereinafter referred to as image type 1 for ease of description) to the fourth hardware module 140 based on the raw image frames after the predetermined processing by the third hardware module 130. The predetermined processing may include, but is not limited to, black level correction, digital gain, bad pixel correction, chromatic aberration correction, exposure adjustment, white balance, wide dynamic range adjustment, lens shading correction, noise removal, color correction, color space conversion, and other processing.

[0043] Optionally, the fourth hardware module 140 can process the image frame of image type 1 from the third hardware module 130 to obtain an image frame of another image type (hereinafter referred to as image type 2 for ease of description), and output the image frame of image type 2 to the fifth hardware module 150. Image type 1 and image type 2 can each be a specific YUV type. It is understood that Y represents luminance, and U and V both represent chrominance.

[0044] Optionally, the fifth hardware module 150 may include a neural network accelerator. The fifth hardware module 150 may use a visual perception algorithm to process the image frame of image type 2 from the fourth hardware module 140 to obtain a visual perception result. For example, the fifth hardware module 150 may perform target detection on the image frame of image type 2 from the fourth hardware module 140 to obtain an image frame of image type 2 with multiple detection frames. The image frame of image type 2 with multiple detection frames may be used as the processing result of the image processing pathway on the original image frame. The processing result of the image processing pathway on the original image frame may be used for subsequent processing. For example, the image processing pathway may be located in a hardware system circuit related to the driving control of the vehicle, and the processing result of the image processing pathway on the original image frame may be used for the vehicle's automatic driving, advanced assisted driving, etc. In one example, the hardware system circuit may include a system on chip (SoC).

[0045] It's important to note that during the image processing pipeline's processing of raw image frames, hardware modules within the pipeline may experience functional safety errors, which can have adverse effects. For example, functional safety errors within hardware modules within the pipeline can impact the reliability of the hardware system circuitry. Therefore, it's necessary to implement measures to minimize the adverse effects of functional safety errors within hardware modules within the pipeline.

[0046] Exemplary Systems

[0047] In order to minimize the adverse effects caused by functional safety errors in hardware modules in the image processing path, a sixth hardware module 210 may be introduced as shown in FIG2 . The sixth hardware module 210 may also be called a message network module.

[0048] In an embodiment of the present disclosure, a first image frame of a first image type can be processed through an image processing pathway to obtain a second image frame of a second image type. Furthermore, during the processing of the first image frame through the image processing pathway, error information about a hardware module in the image processing pathway that has a functional safety error can be obtained through the sixth hardware module 210. Based on the error information obtained by the sixth hardware module 210, a reliability assessment can be performed on the second image frame. Based on the reliability assessment result of the second image frame, a suitable post-processing method can be used to post-process the second image frame, thereby minimizing the adverse effects of the functional safety error of the hardware module in the image processing pathway.

[0049] Exemplary Methods

[0050] FIG3 is a flowchart of a method for processing an image frame based on functional safety according to some exemplary embodiments of the present disclosure. The method shown in FIG3 may include steps 310, 320, 330, and 340.

[0051] Step 310 : During the process of converting the image format of a first image frame of a first image type into a second image type, error information of a hardware module having a functional safety error is determined.

[0052] Prior to step 310, a first image frame of a first image type may be acquired. Optionally, the first image frame of the first image type may be a raw image frame captured by the image sensor mentioned above. Thus, the first image type may include, but is not limited to, RGGB, BGGR, GBRG, RGBIR, and the like, which are not enumerated here. It is understood that R represents red, G represents green, B represents blue, and IR represents infrared.

[0053] In step 310, the first image frame of the first image type may be processed by the image processing pathway to achieve image format conversion from the first image type to the second image type, thereby obtaining a second image frame of the second image type. Alternatively, the second image frame of the second image type may be the result of processing the original image frame by the image processing pathway mentioned above. Thus, the second image type may be the image type 2 mentioned above.

[0054] In addition, in step 310, during the process of processing the first image frame through the image processing path, error information of the hardware module having a functional safety error in the image processing path can be obtained through the sixth hardware module 210 in FIG. 2 .

[0055] Optionally, each hardware module in the image processing path may perform error detection on itself, obtain error information, and report the obtained error information to the sixth hardware module 210 .

[0056] Optionally, the number of hardware modules that experience a functional safety error may be one, two, three, or more than three, and is not listed here. In one example, the number of hardware modules that experience a functional safety error may be five, including the first hardware module 110, the second hardware module 120, the third hardware module 130, the fourth hardware module 140, and the fifth hardware module 150 in FIG. 2 .

[0057] Optionally, the error information of a hardware module experiencing a functional safety error may also be referred to as FUSA information. The error information of a hardware module experiencing a functional safety error may include, but is not limited to, the error type and error severity level. Error types may include, but are not limited to, transmission error types, power supply error types, and clock error types. Error severity levels may include, but are not limited to, L1, L2, L3, L4, and L5.

[0058] Assuming that the autonomous driving system or Advanced Driver Assistance System (ADAS) is referred to as the system, the definitions of levels L1 to L5 can be as follows:

[0059] (1) L1 level: negligible error, no impact on the system;

[0060] (2) L2 level: minor error, system alarm can be issued;

[0061] (3) Level 3: General error, requiring downgrade of some driving functions;

[0062] (4) L4 level: less serious error, requiring self-recovery;

[0063] (5) L5 level: serious error, the system needs to be shut down and enter the safe island state.

[0064] Step 320 : Determine a reliability evaluation result of the second image frame converted to the second image type based on the error information.

[0065] In step 320, based on the error information of the hardware module experiencing the functional safety error, a reliability assessment can be performed on the second image frame to obtain a reliability assessment result for the second image frame. Optionally, the reliability assessment result can be in numerical form, where a larger numerical value indicates a higher reliability of the second image frame, and a smaller numerical value indicates a lower reliability of the second image frame. Alternatively, the reliability assessment result can be in a binary classification form, where a classification of 1 indicates that the reliability of the second image frame meets the requirements, and a classification of 0 indicates that the reliability of the second image frame does not meet the requirements.

[0066] Step 330 : Determine the data attribute of the second image frame based on the reliability evaluation result of the second image frame.

[0067] Optionally, the data attribute of the second image frame may be used to indicate whether the second image frame is suitable for a preset driving task. The preset driving task may include, but is not limited to, an autonomous driving task, an advanced driver assistance task, and the like.

[0068] If the reliability assessment result of the second image frame is in numerical form, then when the numerical value is greater than a preset value, the data attribute used to characterize that the second image frame is suitable for the preset driving task can be determined; when the numerical value is less than or equal to the preset value, the data attribute used to characterize that the second image frame is not suitable for the preset driving task can be determined.

[0069] If the reliability assessment result of the second image frame is in a binary classification form, then when the classification is 1, the data attributes used to characterize that the second image frame is suitable for the preset driving task can be determined; when the classification is 0, the data attributes used to characterize that the second image frame is not suitable for the preset driving task can be determined.

[0070] Of course, the method of determining data attributes is not limited to this. For the sake of clarity, examples are given below.

[0071] Step 340 : Post-process the second image frame in a post-processing manner adapted to the data attributes.

[0072] If the data attributes indicate that the second image frame is suitable for the preset driving task, the second image frame may be post-processed using the first type of post-processing method. For example, the second image frame may be applied to the preset driving task.

[0073] If the data attribute indicates that the second image frame is not suitable for the preset driving task, a second type of post-processing method different from the first type of post-processing method may be used to post-process the second image frame. For example, the second image frame may be prohibited from being used for the preset driving task.

[0074] In an embodiment of the present disclosure, during the process of processing a first image frame of a first image type through an image processing pathway to obtain a second image frame of a second image type, error information of a hardware module in the image processing pathway that has experienced a functional safety error can be determined. Based on the error information of the hardware module that has experienced the functional safety error, a reliability assessment can be performed on the second image frame to obtain a reliability assessment result for the second image frame, thereby determining the data attributes of the second image frame. Based on the data attributes of the second image frame, the second image frame can be post-processed using an appropriate post-processing method. It should be noted that the data attributes of the second image frame are closely related to the error information of the hardware module that has experienced the functional safety error. Accordingly, the post-processing method adopted for the second image frame is also closely related to the error information of the hardware module that has experienced the functional safety error. In other words, the embodiment of the present disclosure collects functional safety information of each hardware module in the image processing pathway as a whole and performs a decision analysis based on the whole, thereby ensuring the rationality of the post-processing method adopted for the second image frame. This helps to avoid the adverse effects caused by functional safety errors in the hardware modules in the image processing pathway. For example, the reliability of the hardware system circuit (e.g., system-on-chip) in which the image processing pathway is located can be guaranteed.

[0075] In some optional examples, step 340 of the present disclosure may include:

[0076] In response to the data attribute indicating that the second image frame is suitable for a preset driving task, based on the second image frame, controlling the movement of the movable device where the sensor for acquiring the first image frame is located;

[0077] In response to the data attribute indicating that the second image frame is not suitable for the preset driving task, the second image frame is discarded.

[0078] Optionally, the sensor for capturing the first image frame may be the image sensor described above, such as a camera. The movable device where the sensor for capturing the first image frame resides may include, but is not limited to, a vehicle, an airplane, a train, a ship, etc., which are not listed here one by one.

[0079] If the data attributes indicate that the second image frame is suitable for a preset driving task, the movable device can be controlled based on the second image frame. For example, if the second image frame determines that an obstacle is present a certain distance ahead of the vehicle, the vehicle can be decelerated or braked to avoid a collision. For another example, if the second image frame determines that the vehicle has veered from its current lane, the vehicle's direction of movement can be controlled to maintain it within that lane. In this way, by referencing visual information, the movable device can be properly controlled to ensure its reliability and safety.

[0080] If the data attributes indicate that the second image frame is unsuitable for the preset driving task, the second image frame can be directly discarded. For example, the second image frame can be directly deleted. This can prevent the second image frame from occupying storage space and from affecting the driving reliability and safety of the mobile device.

[0081] FIG4 is a flow chart illustrating a method for determining a reliability assessment result of a second image frame according to some exemplary embodiments of the present disclosure. The method shown in FIG4 may include steps 410, 420, 430, and 440. Optionally, a combination of steps 410 to 440 may be used as an optional implementation of step 320 of the present disclosure.

[0082] Step 410 : Determine the number of error occurrences corresponding to each of a plurality of preset error levels based on the error information.

[0083] Optionally, the multiple preset error levels may include the above-mentioned L1 level, L2 level, L3 level, L4 level, and L5 level.

[0084] Optionally, the error information may include multiple records, for example, R records, each of which may include an error type, an error severity level, and a module identifier of a hardware module. By counting the number of records with level L1 in the R records, the number of error occurrences corresponding to level L1 can be determined. In an example, if R records are 20 records, and 3 of the 20 records contain level L1, then the number of error occurrences corresponding to level L1 is 3. In a similar manner, the number of error occurrences corresponding to level L2, level L3, level L4, and level L5 can be determined.

[0085] Step 420 : Determine reliability scores corresponding to the plurality of preset error levels based on the number of error occurrences corresponding to the plurality of preset error levels.

[0086] Optionally, an objective function may be pre-set in which the independent variable is the number of errors, the dependent variable is the reliability score, and the independent variable and the dependent variable are negatively correlated. The objective function may include, but is not limited to, an inverse proportional function, an exponential function with a base greater than 0 and less than 1, and the like.

[0087] In step 420, for any preset error level among the plurality of preset error levels, the reliability score value corresponding to the preset error level can be obtained by substituting the number of error occurrences corresponding to the preset error level as the value of the independent variable into the objective function for calculation.

[0088] Step 430 : Determine the level weight corresponding to each of the plurality of preset error levels based on the number of error occurrences corresponding to each of the plurality of preset error levels.

[0089] In some optional implementations of the present disclosure, as shown in FIG. 5 , step 430 may include step 4301 , step 4303 , and step 4305 .

[0090] Step 4301: for any preset error level among a plurality of preset error levels, determine a preset number range to which the number of error occurrences corresponding to the preset error level belongs.

[0091] Optionally, a plurality of preset number ranges may be pre-set, such as (0, c1], (c1, c2], (c2, c3], (c3, c4], etc.

[0092] In step 4301, for any preset error level among multiple preset error levels, the number of error occurrences corresponding to the preset error level can be compared one by one with multiple preset number ranges to determine the preset number range to which the number of error occurrences corresponding to the preset error level belongs.

[0093] Step 4303: Determine a weight adjustment method that is suitable for a preset number range to which the number of error occurrences corresponding to the preset error level belongs.

[0094] Optionally, a correspondence between multiple preset number ranges and multiple weight adjustment methods can be pre-set. The weight adjustment method can be used to indicate a weight adjustment rule, such as keeping the weight unchanged, adjusting the weight upward by a certain percentage, adjusting the weight downward by a certain percentage, etc.

[0095] In step 4303, based on the pre-set correspondence, the weight adjustment method corresponding to the preset number range determined in step 4301 can be determined, and the determined weight adjustment method can be considered as the weight adjustment method adapted to the preset number range.

[0096] Step 4305: Adjust the preset weight corresponding to the preset error level according to the determined weight adjustment method to obtain the level weight corresponding to the preset error level.

[0097] Optionally, a correspondence between multiple preset error levels and multiple preset weights can be pre-set. In one example, the preset weight corresponding to the L1 level can be expressed as k1, the preset weight corresponding to the L2 level can be expressed as k2, the preset weight corresponding to the L3 level can be expressed as k3, the preset weight corresponding to the L4 level can be expressed as k4, and the preset weight corresponding to the L5 level can be expressed as k5, where k1<k2<k3<k4<k5.

[0098] In step 4305, the preset weights may be adjusted according to the weight adjustment method determined in step 4303 to obtain corresponding level weights.

[0099] In an example, the number of errors corresponding to the L1 level is 10 times, which belongs to the preset number range of (c3, c4]. The weight adjustment method adapted to the preset number range of (c3, c4) is used to indicate that the weight is adjusted upward by 20%, then 1.2k1 can be used as the level weight corresponding to the L1 level.

[0100] In another example, the number of errors corresponding to the L2 level is 5 times, which belongs to the preset number range of (c2, c3]. The weight adjustment method adapted to the preset number range of (c2, c3) is used to indicate that the weight is adjusted upward by 15%, then 1.15k2 can be used as the level weight corresponding to the L2 level.

[0101] In the embodiment shown in FIG5 , for any of the multiple preset error levels, the preset weight corresponding to the preset error level can be adjusted using an appropriate weight adjustment method with reference to the number of error occurrences corresponding to the preset error level. For example, for a lower preset error level, if the number of error occurrences is high, the preset weight corresponding to the preset error level can be appropriately amplified to obtain the corresponding level weight. This helps ensure the rationality of the level weight corresponding to the preset error level.

[0102] Step 440 : Using the level weights corresponding to the multiple preset error levels, weight the reliability score values ​​corresponding to the multiple preset error levels to obtain a reliability evaluation result of the second image frame.

[0103] In step 440, the level weights corresponding to the multiple preset error levels can be used to perform a weighted average on the reliability score values ​​corresponding to the multiple preset error levels to obtain a reliability assessment score of the second image frame. The reliability assessment score of the second image frame can be used as the reliability assessment result of the second image frame.

[0104] In an example, the level weight and reliability score corresponding to the L1 level are expressed as k1' and S1, respectively; the level weight and reliability score corresponding to the L2 level are expressed as k2' and S2, respectively; the level weight and reliability score corresponding to the L3 level are expressed as k3' and S3, respectively; the level weight and reliability score corresponding to the L4 level are expressed as k4' and S4, respectively; the level weight and reliability score corresponding to the L5 level are expressed as k5' and S5, respectively; and the reliability assessment score of the second image frame is expressed as S. Then:

[0105] It should be noted that the method for determining the level weights corresponding to each of the multiple preset error levels is not limited to the embodiment shown in FIG5 . For example, similar to the method for determining the reliability score values ​​corresponding to each of the multiple preset error levels, in order to determine the level weights, a function can be introduced in which the independent variable is the number of error occurrences and the dependent variable is the level weight, and the independent variable and the dependent variable are positively correlated, such as a direct proportional function.

[0106] In the disclosed embodiments, the number of error occurrences can be used as a reference factor to determine the reliability scores and level weights corresponding to multiple preset error levels. Weighted processing can then be performed to efficiently and quickly determine the reliability assessment result for the second image frame. In this way, the reliability assessment result for the second image frame is determined by comprehensively considering the number of functional safety errors of different error levels, enabling a comprehensive assessment of the impact of functional safety errors on the image frame, which helps ensure the rationality of the determined reliability assessment result.

[0107] FIG6 is a flowchart illustrating a method for determining a reliability assessment result of a second image frame according to another exemplary embodiment of the present disclosure. The method shown in FIG6 may include steps 610, 620, and 630. Alternatively, a combination of steps 620 and 630 may be used as an alternative implementation of step 320 of the present disclosure.

[0108] Step 610: Obtain a pre-trained reliability assessment model.

[0109] Optionally, before executing the functional safety-based image frame processing method provided by the embodiment of the present disclosure, a large amount of sample data can be used to perform model training to obtain a reliability assessment model. Each sample data may include an image frame, error information of a hardware module in an image processing path, and annotation information used to characterize the degree of influence of the error information on the reliability of the image frame. In some embodiments, the degree of influence may be in the form of an impact score, where the larger the value of the impact score, the greater the impact of the error information on the reliability of the image frame, and the smaller the value of the impact score, the smaller the impact of the error information on the reliability of the image frame. In other embodiments, the degree of influence may be in the form of an impact level, where the higher the impact level, the greater the impact of the error information on the reliability of the image frame, and the lower the impact level, the smaller the impact of the error information on the reliability of the image frame. In yet other embodiments, the degree of influence may also be in the form of an impact percentage. In this way, the reliability assessment model obtained through training can effectively assess the degree of influence of a given error information on the reliability of a given image frame.

[0110] Step 620: Evaluate the impact of the error information on the reliability of the second image frame using a reliability evaluation model.

[0111] In step 620, the second image frame and the error information determined in step 320 can be used as input data of the reliability assessment model. The reliability assessment model can operate on the error information based on the input data to evaluate the degree of impact of the error information determined in step 320 on the reliability of the second image frame.

[0112] Step 630: Obtain a reliability evaluation result of the second image frame based on the impact degree.

[0113] In some embodiments, the degree of influence obtained in step 620 can be in the form of an influence score. If the influence score is greater than a preset score, a reliability assessment result can be obtained to characterize that the reliability of the second image frame does not meet the requirements. If the influence score is less than or equal to the preset score, a reliability assessment result can be obtained to characterize that the reliability of the second image frame meets the requirements.

[0114] In other embodiments, the degree of influence obtained in step 620 may be in the form of an influence level. If the influence level is higher than a preset level, a reliability evaluation result may be obtained to characterize that the reliability of the second image frame does not meet the requirements. If the influence level is lower than or equal to the preset level, a reliability evaluation result may be obtained to characterize that the reliability of the second image frame meets the requirements.

[0115] In the embodiment of the present disclosure, the reliability assessment model is trained based on a large amount of sample data. The reliability assessment model can objectively and accurately assess the impact of given error information on the reliability of a given image frame. Therefore, the reliability assessment result of the second image frame is obtained based on the reliability assessment model, which can better ensure the objectivity and accuracy of the reliability assessment result.

[0116] FIG7 is a flow chart illustrating a method for determining data attributes of a second image frame according to some exemplary embodiments of the present disclosure. The method shown in FIG7 may include steps 710, 720, 730, 740, and 750. Alternatively, a combination of steps 710 to 750 may be used as an alternative implementation of step 330 of the present disclosure.

[0117] Step 710 : Determine a first reliability evaluation score and a second reliability evaluation score based on respective reliability evaluation results of two second image frames; wherein the two second image frames are obtained through different image processing pathways.

[0118] Optionally, the first image frame may be processed respectively through two image processing pathways to obtain two second image frames corresponding one-to-one to the two image processing pathways.

[0119] In one example, a first image frame is image frame P1, and two image processing pathways are image processing pathway W1 and image processing pathway W2. A second image frame obtained by processing image frame P1 via image processing pathway W1 is image frame P2, and a second image frame obtained by processing image frame P1 via image processing pathway W2 is image frame P3. Therefore, error information of a hardware module in image processing pathway W1 that experienced a functional safety error during processing of image frame P1 by image processing pathway W1 can be determined, and a reliability assessment result for image frame P2 can be determined based on this error information. Similarly, error information of a hardware module in image processing pathway W2 that experienced a functional safety error during processing of image frame P1 by image processing pathway W1 can be determined, and a reliability assessment result for image frame P3 can be determined based on this error information. The reliability assessment results for image frame P2 and image frame P3 can each include a reliability assessment score. Therefore, the reliability assessment score in the reliability assessment result for image frame P2 can be used as a first reliability assessment score, and the reliability assessment score in the reliability assessment result for image frame P3 can be used as a second reliability assessment score.

[0120] Of course, the method of determining the first reliability assessment score and the second reliability assessment score is not limited to this. For example, a reliability assessment score can be obtained from the reliability assessment result of image frame P2 and the reliability assessment result of image frame P3 respectively to obtain two reliability assessment scores, and the two reliability assessment scores are normalized. One of the two reliability assessment scores after normalization can be used as the first reliability assessment score, and the other can be used as the second reliability assessment score.

[0121] Step 720: Determine the score difference between the first reliability evaluation score and the second reliability evaluation score.

[0122] In step 720 , the first reliability evaluation score and the second reliability evaluation score may be subtracted, and an absolute value of the subtraction result may be calculated, and the absolute value may be used as the score difference.

[0123] Step 730: Determine a first numerical relationship between the score difference and a preset difference.

[0124] Optionally, the first numerical relationship can be represented by a magnitude relationship between the fractional difference and the preset difference, or the first numerical relationship can be represented by a proportional relationship between the fractional difference and the preset difference.

[0125] Step 740 : Determine a second numerical relationship between the first reliability assessment score and the second reliability assessment score.

[0126] Optionally, the second numerical relationship can be represented by a magnitude relationship between the first reliability evaluation score and the second reliability evaluation score, or the second numerical relationship can be represented by a proportional relationship between the first reliability evaluation score and the second reliability evaluation score.

[0127] Step 750: Determine data attributes of the two second image frames based on the first numerical relationship and the second numerical relationship.

[0128] Still taking the case where the two second image frames are image frame P2 and image frame P3 as an example, assuming that the first reliability evaluation score is S and corresponds to image frame P2, and the second reliability evaluation score is S' and corresponds to image frame P3, then we can have:

[0129] (1) If the first numerical relationship represents that the fractional difference is less than the preset difference, and the second numerical relationship represents that S is greater than S′, then the data attribute of image frame P2 is used to represent that image frame P2 is suitable for the preset driving task, and the data attribute of image frame P3 is used to represent that image frame P3 is not suitable for the preset driving task;

[0130] (2) If the first numerical relationship represents that the fractional difference is less than the preset difference, and the second numerical relationship represents that S is less than S′, then the data attribute of image frame P3 is used to represent that image frame P3 is suitable for the preset driving task, and the data attribute of image frame P2 is used to represent that image frame P2 is not suitable for the preset driving task;

[0131] (3) If the first numerical relationship represents that the score difference is greater than or equal to the preset difference, regardless of the second numerical relationship, the data attributes of image frame P2 are used to represent that image frame P2 is not suitable for the preset driving task, and the data attributes of image frame P3 are used to represent that image frame P3 is not suitable for the preset driving task.

[0132] In embodiments of the present disclosure, two image processing pathways with identical structures and parameters can be used to process a first image frame to obtain two second image frames. A first reliability assessment score corresponding to one of the second image frames and a second reliability assessment score corresponding to the other second image frame can be determined. The difference between the first and second reliability assessment scores can be used to indicate the magnitude of the difference in processing results for the first image frame by the two image processing pathways. A first numerical relationship between the score difference and a preset difference value can be used to indicate whether the magnitude of the difference is within a reasonable range. A second numerical relationship between the first and second reliability assessment scores can be used to indicate the relative reliability of the two second image frames. It should be noted that, in theory, the processing results of the two image processing pathways for the first image frame are substantially consistent. Therefore, based on the first numerical relationship, it is possible to verify whether the actual processing results of the two image processing pathways for the first image frame are consistent with the theoretical situation, thereby reasonably determining data attributes. Furthermore, based on the second numerical relationship, the relative reliability of different second image frames can be applied to the data attribute determination process, which considers a wider range of factors and helps ensure the rationality of the determined data attributes.

[0133] Figure 8 is a flowchart of a method for updating error information provided by some exemplary embodiments of the present disclosure. The method shown in Figure 8 may include steps 810, 820, 830, and 840. Optionally, the method shown in Figure 8 may be performed after step 310 of the present disclosure.

[0134] Step 810 : Determine the target module in the image processing path where the hardware module where the functional safety error occurs is located.

[0135] Assuming that the error information determined in step 310 includes error information of first hardware module 110, error information of second hardware module 120, and error information of fourth hardware module 140 in FIG2 , the target module determined in step 810 may include at least one of the following: third hardware module 130 in FIG2 , and fifth hardware module 150 in FIG2 . In other words, the target module may include at least some of the remaining hardware modules in the image processing pathway excluding first hardware module 110, second hardware module 120, and fourth hardware module 140.

[0136] Step 820: Send an error reporting instruction to the target module.

[0137] Taking the case where the target module includes the third hardware module 130 and the fifth hardware module 150 as an example, in step 820, the sixth hardware module 210 in Figure 2 can send error reporting instructions to the third hardware module 130 and the fifth hardware module 150 respectively.

[0138] Step 830: Determine the response information of the target module to the error reporting instruction.

[0139] Optionally, during the process of processing the first image frame through the image processing pathway, each hardware module in the image processing pathway may detect whether it has a functional safety error, and if so, may generate and store corresponding record information. The composition of the record information can refer to the relevant description above.

[0140] After the third hardware module 130 receives the error reporting instruction, if the third hardware module 130 stores record information, it may include this record information in a response message and send it to the sixth hardware module 210. If the third hardware module 130 does not store record information, the third hardware module 150 may directly send response information to the sixth hardware module 210, indicating that no functional safety error has occurred in the third hardware module 130. In this way, the sixth hardware module 210 can receive the response information from the third hardware module 130.

[0141] Step 840: Based on the response information, update the error information of the hardware module where the functional safety error occurs.

[0142] Assuming that the response information of the third hardware module 130 and the fifth hardware module 150 to the error reporting instruction both include some record information, the record information in the response information of the third hardware module 130 and the fifth hardware module 150 to the error reporting instruction can be fused with the error information determined in step 310, and the fusion result can be used as the update result of the error information determined in step 310.

[0143] In one example, the error information determined in step 310 is shown in Table 1 below:

[0144] Table 1 Error information

[0145] Assuming that the response information sent by the third hardware module 130 includes the power error type and the L3 level, and the error information in the response information sent by the fifth hardware module 150 includes the clock error type and the L4 level, Table 1 can be updated to obtain the following Table 2:

[0146] Table 2 Error information update results

[0147] Table 2 may be used as an update result of the error information determined in step 310 .

[0148] In embodiments of the present disclosure, if the error information determined in step 310 only includes error information corresponding to some hardware modules in the image processing pathway, an error reporting instruction may be sent to at least some of the remaining modules in the image processing pathway, and the error information may be updated based on the response information from at least some of the remaining modules to the error reporting instruction. This helps ensure the integrity of the error information used to determine the reliability assessment result of the second image frame, avoids omissions of error information, and thus effectively ensures the accuracy of the reliability assessment result.

[0149] In some optional examples, as shown in FIG9 , a first image frame of a first image type may be processed through an image processing pathway to obtain a second image frame of a second image type. Furthermore, during the processing of the first image frame through the image processing pathway, error information of a hardware module in the image processing pathway that has a functional safety error may be obtained through the sixth hardware module 210. For example, during the processing of the first hardware module 110, error information may be obtained for the first hardware module 110; during the processing of the second hardware module 120, error information may be obtained for the second hardware module 120; and during the processing of the fifth hardware module 150, error information may be obtained for the fifth hardware module 150.

[0150] After the image processing path obtains the second image frame, the error information obtained by the sixth hardware module 210 can be summarized. Based on the second image frame and the summarized error information, the degree of influence of the summarized error information on the reliability of the second image frame can be evaluated through a reliability assessment model. Based on the degree of influence, a reliability assessment result of the second image frame can be obtained. The reliability assessment result of the second image frame can be used to characterize whether the reliability of the second image frame meets the requirements. If the reliability of the second image frame meets the requirements, the driving control of the movable device can be performed based on the second image frame to ensure the reliability of automatic driving or advanced assisted driving. If the reliability of the second image frame does not meet the requirements, the second image frame can be directly discarded.

[0151] In summary, by adopting the embodiments of the present disclosure, it is not necessary to perform isolated processing on the functional safety errors of each hardware module in the image processing pathway. Instead, the functional safety information of each hardware module in the image processing pathway can be collected as a whole, and a decision analysis can be performed on the whole, thereby ensuring the rationality of the post-processing method adopted for the second image frame, and thus being able to very effectively avoid the adverse effects caused by functional safety errors of the hardware modules in the image processing pathway.

[0152] Exemplary devices

[0153] FIG10 is a schematic diagram of a functional safety-based image frame processing apparatus according to some exemplary embodiments of the present disclosure. The apparatus shown in FIG10 includes a first determination module 1010 , a second determination module 1020 , a third determination module 1030 , and a post-processing module 1040 .

[0154] A first determining module 1010 is configured to determine error information of a hardware module having a functional safety error during a process of converting an image format of a first image frame of a first image type into a second image type;

[0155] A second determining module 1020 is configured to determine a reliability evaluation result of a second image frame converted to a second image type based on the error information determined by the first determining module 1010;

[0156] A third determining module 1030 is configured to determine a data attribute of the second image frame based on the reliability evaluation result of the second image frame determined by the second determining module 1020;

[0157] The post-processing module 1040 is configured to post-process the second image frame in a post-processing manner that matches the data attribute determined by the third determining module 1030 .

[0158] In some optional examples, as shown in FIG11 , the second determining module 1020 includes:

[0159] A first determining submodule 1110 is configured to determine the number of error occurrences corresponding to each of a plurality of preset error levels based on the error information determined by the first determining module 1010;

[0160] The second determining submodule 1120 is configured to determine reliability scores corresponding to the plurality of preset error levels based on the number of error occurrences corresponding to the plurality of preset error levels determined by the first determining submodule 1110;

[0161] The third determining submodule 1130 is configured to determine a level weight corresponding to each of the plurality of preset error levels based on the number of error occurrences corresponding to each of the plurality of preset error levels in the first determining submodule 1110;

[0162] The weighting submodule 1140 is used to weight the reliability score values ​​corresponding to the multiple preset error levels determined by the second determination submodule 1120 using the level weights corresponding to the multiple preset error levels determined by the third determination submodule 1130 to obtain the reliability evaluation result of the second image frame.

[0163] In some optional examples, the third determining submodule 1130 includes:

[0164] A first determining subunit is configured to determine, for any preset error level among a plurality of preset error levels, a preset number range to which the number of error occurrences corresponding to the preset error level belongs;

[0165] A second determining subunit is configured to determine a weight adjustment method that is compatible with the preset number range determined by the first determining subunit;

[0166] The adjusting subunit is configured to adjust the preset weight corresponding to the preset error level according to the weight adjustment method determined by the second determining subunit to obtain the level weight corresponding to the preset error level.

[0167] In some optional examples, the second determining module 1020 includes:

[0168] An evaluation submodule, configured to evaluate the degree of influence of the error information determined by the first determination module 1010 on the reliability of the second image frame using a pre-trained reliability evaluation model;

[0169] The acquisition submodule is used to obtain a reliability evaluation result of the second image frame based on the impact degree evaluated by the evaluation submodule.

[0170] In some optional examples, as shown in FIG12 , the third determining module 1030 includes:

[0171] A fourth determination submodule 1210 is configured to determine a first reliability evaluation score and a second reliability evaluation score based on respective reliability evaluation results of the two second image frames, wherein the two second image frames are obtained through different image processing pathways;

[0172] The fifth determining submodule 1220 is configured to determine a score difference between the first reliability evaluation score and the second reliability evaluation score determined by the fourth determining submodule 1210;

[0173] a sixth determining submodule 1230, configured to determine a first numerical relationship between the score difference determined by the fifth determining submodule 1220 and a preset difference;

[0174] a seventh determining submodule 1240 , configured to determine a second numerical relationship between the first reliability evaluation score and the second reliability evaluation score determined by the fourth determining submodule 1210 ;

[0175] The eighth determining submodule 1250 is configured to determine data attributes of the two second image frames based on the first numerical relationship determined by the sixth determining submodule 1230 and the second numerical relationship determined by the seventh determining submodule 1240 .

[0176] In some optional examples, as shown in FIG13 , the apparatus provided by the embodiments of the present disclosure further includes:

[0177] A fourth determining module 1310 is configured to determine a target module in the image processing path where the hardware module is located after the first determining module 1010 determines the error information;

[0178] A sending module 1320 is configured to send an error reporting instruction to the target module determined by the fourth determining module 1310;

[0179] A fifth determining module 1330 is configured to determine response information of the target module determined by the fourth determining module 1310 to the error reporting instruction sent by the sending module 1320;

[0180] The updating module 1340 is configured to update the error information determined by the first determining module 1010 based on the response information determined by the fifth determining module 1330 .

[0181] In some optional examples, the post-processing module 1040 includes:

[0182] a first processing submodule for, in response to the data attribute determined by the third determining module 1030 to indicate that the second image frame is suitable for a preset driving task, controlling the movement of a movable device including a sensor for capturing the first image frame based on the second image frame;

[0183] The second processing submodule is configured to discard the second image frame in response to the third determining module 1030 determining that the data attribute indicates that the second image frame is not suitable for the preset driving task.

[0184] In the device of the present disclosure, the various optional embodiments, optional implementation methods and optional examples disclosed above can be flexibly selected and combined as needed to achieve corresponding functions and effects, and the present disclosure does not list them one by one.

[0185] The beneficial technical effects corresponding to the exemplary embodiment of this device can be found in the corresponding beneficial technical effects of the above exemplary method part, which will not be repeated here.

[0186] Exemplary electronic devices

[0187] FIG. 14 illustrates a block diagram of an electronic device according to an embodiment of the present disclosure. The electronic device 1400 includes one or more processors 1410 and a memory 1420 .

[0188] The processor 1410 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 1400 to perform desired functions.

[0189] The memory 1420 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), a hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 1410 may execute the one or more computer program instructions to implement the methods of the various embodiments of the present disclosure described above and / or other desired functions.

[0190] In one example, the electronic device 1400 may further include an input device 1430 and an output device 1440 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0191] The input device 1430 may also include, for example, a keyboard, a mouse, and the like.

[0192] The output device 1440 can output various information to the outside, and may include, for example, a display, a speaker, a printer, a communication network and its connected remote output device, etc.

[0193] Of course, for simplicity, FIG14 only shows some of the components related to the present disclosure in the electronic device 1400, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 1400 may further include any other appropriate components depending on the specific application.

[0194] Exemplary computer program products and computer-readable storage media

[0195] In addition to the above-mentioned methods and devices, an embodiment of the present disclosure may also be a computer program product, which includes computer program instructions that, when executed by a processor, enable the processor to perform the steps of the method according to various embodiments of the present disclosure described in the above-mentioned "Exemplary Method" section of this specification.

[0196] The computer program product may be written in any combination of one or more programming languages ​​to implement the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, and conventional procedural programming languages ​​such as C or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user's computing device, as a stand-alone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0197] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, causes the processor to execute the steps of the method according to various embodiments of the present disclosure described in the above “Exemplary Method” section of this specification.

[0198] The computer-readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can, for example, include but is not limited to a system, device or component of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0199] The basic principles of the present disclosure have been described above in conjunction with specific embodiments. However, the advantages, strengths, and effects mentioned in this disclosure are merely illustrative and non-limiting, and should not be construed as necessarily possessed by each embodiment of the present disclosure. The specific details disclosed above are provided for illustrative purposes and to facilitate understanding, and are not intended to be limiting. The above details do not limit the present disclosure to necessarily being implemented using the above specific details.

[0200] Those skilled in the art may make various changes and modifications to the present disclosure without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present disclosure and their equivalents, the present disclosure is intended to include these modifications and variations.

Claims

1. A functional safety-based image frame processing method, comprising: During a process of converting an image format of a first image frame of a first image type into a second image type, determining error information of a hardware module in which a functional safety error occurs; determining a reliability assessment result of a second image frame converted to the second image type based on the error information; determining a data attribute of the second image frame based on a reliability evaluation result of the second image frame; The second image frame is post-processed according to a post-processing method adapted to the data attributes.

2. The method according to claim 1, wherein The determining, based on the error information, a reliability evaluation result of the second image frame converted to the second image type includes: Based on the error information, determining the number of error occurrences corresponding to each of a plurality of preset error levels; Determining reliability score values corresponding to each of the plurality of preset error levels based on the number of error occurrences corresponding to each of the plurality of preset error levels; Determining a level weight corresponding to each of the plurality of preset error levels based on the number of error occurrences corresponding to each of the plurality of preset error levels; The reliability score values corresponding to each of the plurality of preset error levels are weighted by using the level weights corresponding to each of the plurality of preset error levels to obtain a reliability evaluation result of the second image frame.

3. The method according to claim 2, wherein: The determining, based on the error occurrence counts corresponding to the plurality of preset error levels, the level weights corresponding to the plurality of preset error levels, includes: For any one of the plurality of preset error levels, determining a preset number range to which the number of error occurrences corresponding to the preset error level belongs; Determining a weight adjustment method that is suitable for the preset number range; According to the weight adjustment method, the preset weight corresponding to the preset error level is adjusted to obtain the level weight corresponding to the preset error level.

4. The method according to claim 1, wherein The determining, based on the error information, a reliability evaluation result of the second image frame converted to the second image type includes: evaluating, by a pre-trained reliability evaluation model, the degree of influence of the error information on the reliability of the second image frame; Based on the impact degree, a reliability evaluation result of the second image frame is obtained.

5. The method according to claim 1, wherein The determining the data attribute of the second image frame based on the reliability evaluation result of the second image frame includes: determining a first reliability evaluation score and a second reliability evaluation score based on the respective reliability evaluation results of the two second image frames; wherein the two second image frames are obtained through different image processing pathways; determining a score difference between the first reliability assessment score and the second reliability assessment score; Determining a first numerical relationship between the fractional difference and a preset difference; determining a second numerical relationship between the first reliability assessment score and the second reliability assessment score; Based on the first numerical relationship and the second numerical relationship, data attributes of each of the two second image frames are determined.

6. The method according to claim 1, further comprising: After determining the error information, determining a target module in the image processing path where the hardware module is located; Sending an error reporting instruction to the target module; Determining response information of the target module to the error reporting instruction; The error information is updated based on the response information.

7. The method according to any one of claims 1 to 6, wherein: The post-processing of the second image frame according to a post-processing method adapted to the data attribute includes: In response to the data attribute indicating that the second image frame is suitable for a preset driving task, based on the second image frame, controlling the movement of a movable device where a sensor for acquiring the first image frame is located; In response to the data attribute indicating that the second image frame is not suitable for the preset driving task, the second image frame is discarded.

8. An image frame processing device based on functional safety, comprising: a first determining module, configured to determine error information of a hardware module having a functional safety error during a process of converting an image format of a first image frame of a first image type into a second image type; a second determining module, configured to determine a reliability evaluation result of a second image frame converted to the second image type based on the error information determined by the first determining module; a third determining module, configured to determine a data attribute of the second image frame based on the reliability evaluation result of the second image frame determined by the second determining module; A post-processing module is used to post-process the second image frame according to a post-processing method that is compatible with the data attribute determined by the third determination module.

9. A computer-readable storage medium storing a computer program, wherein the computer program is used to execute the functional safety-based image frame processing method according to any one of claims 1 to 7.

10. An electronic device, comprising: processor; a memory for storing instructions executable by the processor; The processor is configured to read the executable instructions from the memory and execute the executable instructions to implement the functional safety-based image frame processing method according to any one of claims 1 to 7.

Images