System-on-chip-based image processing method, apparatus, device, and storage medium

Abstract

To provide a system-on-chip-based image processing method, apparatus, device, and a storage medium for improving the safety of intelligent driving by effectively eliminating functional safety (FuSa) abnormalities (to modules) and avoiding an occurrence of a problem of the entire video processing system (VPS).SOLUTION: In an intelligent driving system, the method according to the present invention includes: determining abnormality information generated during processing a first image data frame by a plurality of image processing modules in the system-on-chip and path configuration information corresponding to the plurality of image processing modules; determining an abnormality module that generates the abnormality information and an initial processing policy to address the abnormality information; determining a target processing object module from the plurality of image processing modules on the basis of the abnormality module, the initial processing policy and the path configuration information; processing the target processing object module on the basis of the initial processing policy to acquire the processed target processing object module; and processing a second image data frame by the processed target processing object module.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to intelligent driving technology, and in particular to a system-on-chip image processing method, device, equipment, and storage medium. [Background technology]

[0002] In an intelligent driving system, when an on-board camera captures image data, the image data can be processed by a video processing system (VPS) in an intelligent driving chip (corresponding to a system-on-chip). Typically, the VPS includes multiple image processing modules that can process the images collected by the on-board camera and control the vehicle to perform corresponding actions.

[0003] When a functional safety (FuSa) anomaly occurs in one of the image processing modules during the process of image data processing by multiple image processing modules, the anomaly is typically resolved and recovery processing is performed only on the image processing module where the anomaly occurred, thereby resolving the FuSa anomaly. However, because the image processing modules in the VPS are coupled to each other, processing only the image processing module where the anomaly occurred not only fails to resolve the FuSa anomaly, but may also cause the entire VPS to malfunction, potentially resulting in problems such as a VPS crash. Summary of the Invention [Problem to be solved by the invention]

[0004] Normally, the method of resolving and recovering from an abnormality in an image processing module that has experienced an abnormality not only fails to resolve the FuSa abnormality, but can also cause the entire VPS to become abnormal, potentially resulting in problems such as the VPS crashing. [Means for solving the problem]

[0005] In order to solve the above technical problems, the present disclosure provides an image processing method using a system on a chip, the method comprising: determining abnormality information occurring during the process of processing a first image data frame by a plurality of image processing modules in the system-on-chip and path setting information corresponding to the plurality of image processing modules; determining an abnormal module where the abnormal information occurs and an initial processing policy for handling the abnormal information; determining a target processing module from the plurality of image processing modules based on the abnormal module, the initial processing policy, and the path setting information; processing the target processing target module based on the initial processing policy; and processing the second image data frame with the processed target processing object module.

[0006] In a second aspect of the present invention, there is provided an image processing device in the form of a system on a chip, the device comprising: a first determination module for determining abnormality information occurring during the process of processing a current image data frame by a plurality of image processing modules in the system-on-chip and path setting information corresponding to the plurality of image processing modules; a second determination module for determining an anomaly module where the anomaly information occurs and an initial processing policy for handling the anomaly information; a third determination module for determining a target processing module from the plurality of image processing modules based on the abnormal module, the initial processing policy, and the path setting information; a first processing module for processing the target processing target module based on the initial processing policy; and a second processing module for processing the next image data frame by the processed target processing module.

[0007] In a third aspect of the present disclosure, there is provided a computer-readable storage medium, which stores a computer program for executing the image processing method by the system-on-chip of the first aspect described above.

[0008] In a fourth aspect of the present disclosure, there is provided an electronic device, the electronic device comprising: a processor; a memory for storing processor-executable instructions; The processor reads and executes executable instructions from the memory, thereby realizing the image processing method using the system-on-chip according to the first aspect. [Effects of the Invention]

[0009] In an embodiment of the present disclosure, abnormality information generated during processing of a first image data frame by multiple image processing modules in a system-on-chip and path setting information corresponding to the multiple image processing modules are determined, and an abnormal module and an initial processing policy are determined based on the abnormality information. A target processing module is then determined from the multiple image processing modules based on the abnormal module, the initial policy, and the path setting information. That is, the process of determining the target processing module comprehensively takes into account the abnormal path module, the coupling relationship between the abnormal path module and other path modules, and abnormalities in actual image processing scenarios. Therefore, processing the determined target processing module based on the initial processing policy can effectively resolve the FuSa abnormality, avoid problems such as an abnormality or crash of the entire VPS, and improve the safety of intelligent operation. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a schematic diagram of an intelligent driving chip according to one exemplary embodiment of the present disclosure; [Figure 2] FIG. 1 is a schematic diagram of another intelligent driving chip according to one exemplary embodiment of the present disclosure. [Figure 3]1 is a flowchart of an image processing method by a system-on-chip according to one exemplary embodiment of the present disclosure. [Figure 4] 10 is a flowchart of an image processing method by another system-on-chip according to an exemplary embodiment of the present disclosure. [Figure 5] 10 is a flowchart of yet another system-on-chip image processing method according to an exemplary embodiment of the present disclosure. [Figure 6] 10 is a flowchart of yet another system-on-chip image processing method according to an exemplary embodiment of the present disclosure. [Figure 7] 10 is a flowchart of yet another system-on-chip image processing method according to an exemplary embodiment of the present disclosure. [Figure 8] 10 is a flowchart of yet another system-on-chip image processing method according to an exemplary embodiment of the present disclosure. [Figure 9] 10 is a flowchart of yet another system-on-chip image processing method according to an exemplary embodiment of the present disclosure. [Figure 10] 10 is a flowchart of yet another system-on-chip image processing method according to an exemplary embodiment of the present disclosure. [Figure 11] 10 is a flowchart of yet another system-on-chip image processing method according to an exemplary embodiment of the present disclosure. [Figure 12] 1 is a schematic diagram illustrating the configuration of an image processing device using a system-on-chip according to an exemplary embodiment of the present disclosure. [Figure 13] FIG. 10 is a schematic diagram of the configuration of another system-on-chip image processing device according to an exemplary embodiment of the present disclosure. [Figure 14] FIG. 10 is a schematic diagram of a configuration of yet another system-on-chip image processing device according to an exemplary embodiment of the present disclosure. [Figure 15] FIG. 10 is a schematic diagram of a configuration of yet another system-on-chip image processing device according to an exemplary embodiment of the present disclosure. [Figure 16]FIG. 10 is a schematic diagram of a configuration of yet another system-on-chip image processing device according to an exemplary embodiment of the present disclosure. [Figure 17] FIG. 10 is a schematic diagram of a configuration of yet another system-on-chip image processing device according to an exemplary embodiment of the present disclosure. [Figure 18] FIG. 10 is a schematic diagram of a configuration of yet another system-on-chip image processing device according to an exemplary embodiment of the present disclosure. [Figure 19] 1 is a schematic diagram illustrating the configuration of an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0011] In order to understand the present disclosure, exemplary embodiments of the present disclosure will be described in detail below with reference to the drawings. The described embodiments are only some of the embodiments of the present disclosure, and are not all of the embodiments, and the present disclosure is not limited to the exemplary embodiments.

[0012] The relative arrangement of components and steps, numerical expressions and values ​​set forth in these examples do not limit the scope of the present disclosure unless specifically stated otherwise.

[0013] [Summary of this disclosure] In an intelligent driving system, the driving environment of the vehicle is captured by on-board cameras installed at various positions on the vehicle, the vehicle's driving route is planned based on the processed image data, and the original images output from the on-board cameras must be processed by each image processing module in the VPS so that the vehicle's motors, actuators, etc. are controlled to perform corresponding operations. Therefore, in order to improve the stability and safety of intelligent driving, it is necessary to ensure the quality of the processed image data.

[0014] Currently, the FuSa mechanism is an important capability in the vehicle industry, and the VPS in the intelligent driving chip and each image processing module in the VPS also have the FuSa mechanism. Generally, when multiple image processing modules in the VPS process the original raw image, if an FuSa abnormality occurs in one of the multiple image processing modules, only the internal logic of the image processing module where the FuSa abnormality occurs will be isolated and forced to perform abnormality resolution and recovery processing to resolve the FuSa abnormality.

[0015] However, there is a certain coupling between the upstream and downstream image processing modules in the video path of the VPS. Therefore, if only the image processing module where an abnormality occurs is processed, the entire video path cannot be synchronized. Some FuSa abnormalities that occur require repeated analysis and operation of different image processing modules, or the FuSa abnormality that occurs in the previous image processing module will flow into the subsequent image processing module. This not only leaves the FuSa abnormality unresolved, but also causes the entire VPS to become abnormal, resulting in problems such as a VPS crash, which may ultimately affect the execution efficiency of the in-vehicle system and the safety of intelligent driving.

[0016] 1 is a schematic diagram of an intelligent driving chip according to an exemplary embodiment of the present disclosure. As shown in FIG. 1, the intelligent driving chip 10 includes a VPS 11 and a FuSa abnormality processing module 12. The VPS 11 includes a data collection module 111, a processing module 112, an intelligent driving algorithm module 113, and an interface 114. The processing module 112 includes a Mobile Industry Processor Interface (MIPI) module 1121, a Camera Interface Module (CIM) 1122, an Image Signal Processor (ISP) module 1123, an Image PYraMid (PYM) module 1124, a video codec (CODEC) module 1125, and a Joint Photographic Experts Group (JPEG) Processing Unit (JPU) 1126.

[0017] Here, the input terminal of the data collection module 111 is connected to the output terminal of the vehicle-mounted camera. The processing module 112 is connected between the output terminal of the data collection module 111 and the input terminal of the intelligent driving algorithm module 113. The interface 114 is connected between the processing module 112 and the FuSa abnormality processing module 12.

[0018] The image processing process will now be described by way of example with reference to FIG.

[0019] First, the user inputs a module connection setting operation to the processing module 112 in the VPS 11, and in response to this module connection setting operation, the processing module 112 establishes a video path based on the MIPI module 1121, CIM 1122, ISP module 1123, PYM module 1124, CODEC module 1125, JPU 1126, etc. (In some examples, the video path may include some image processing modules in the processing module 112. For example, the video path may include the MIPI module 1121, CIM 1122, ISP module 1123, PYM module 1124, and JPU 1126 connected in sequence).

[0020] Then, the user inputs an operation mode setting operation to the processing module 112 in the VSP11, and the processing module 112 determines the operation mode setting information in response to this operation mode setting operation, and determines its own processing module for intercepting and processing data from each path module of the video path based on the operation mode setting information.

[0021] Next, the data collection module 111 acquires original image data (raw images) from the output terminal of the vehicle-mounted camera, and outputs the acquired raw images to the input terminal of the processing module 112.

[0022] Next, each path module in the video path of the processing module 112 sequentially processes the input raw images to obtain processed image data, and outputs the processed image data to the intelligent driving algorithm module 113.

[0023] Finally, the intelligent driving algorithm module 113 generates intelligent driving data according to the processed image data, and outputs the intelligent driving data to the intelligent driving system to realize intelligent driving for the vehicle.

[0024] Take the example of a FuSa anomaly occurring in CIM 1122 during the process of each path module in the video path of processing module 112 sequentially processing raw images. The FuSa anomaly processing process is as follows:

[0025] First, the CIM 1122 inputs the abnormality information to the interface 114 .

[0026] Next, the interface 114 receives the anomaly information and transmits the anomaly information to the FuSa anomaly processing module 12 .

[0027] Next, the FuSa abnormality processing module 12 determines that the image processing module in which the FuSa abnormality occurred is CIM1122 based on the received abnormality information, generates a processing operation command (e.g., frame drop or reset operation) for CIM1122, and transmits the processing operation command to CIM1122 via the interface 114.

[0028] Finally, the CIM 1122 receives processing operation commands and forces its internal logic to perform fault resolution and recovery operations in response to the processing operation commands.

[0029] In the embodiment of the present disclosure, since the CIM 1122 is coupled with the MIPI module 1121, the ISP module 1123, the PYM module 1124, the CODEC module 1125, and the JPU 1126, processing only the CIM 1122 may cause repeated analysis and operation of the MIPI module 1121 and / or the ISP module 1123, or the FuSa abnormality may be introduced into other image processing modules downstream of the ISP module 1123 and the CIM 1122. In this case, not only may the FuSa abnormality not be resolved, but the entire VPS may become abnormal, causing problems such as a VPS crash, which may affect the execution efficiency of the in-vehicle system and the safety of intelligent driving.

[0030] In view of the above technical problems, embodiments of the present disclosure provide a method for collaborative processing of FuSa anomalies occurring in any image processing module in the video path by image processing modules in the entire video path. In some examples, all FuSa anomalies in the video path can be analyzed and classified, and corresponding processing measures can be adopted for different anomaly types, realizing collaborative processing between software and hardware in the entire video path, thereby effectively resolving FuSa anomalies and avoiding VPS anomalies. Furthermore, image quality can be guaranteed and the stability and safety of intelligent operation can be improved.

[0031] [Example System] 2 is a schematic diagram of another intelligent driving chip according to an exemplary embodiment of the present disclosure. As shown in FIG. 2, the intelligent driving chip 20 includes a VPS 21 and a FuSa fault processing system 22. The VPS 21 includes a data collection module 211, a processing module 212, an intelligent driving algorithm module 213, and an interface 214. The processing module 212 includes image processing modules such as a MIPI module 2121, a CIM 2122, an ISP module 2123, a PYM module 2124, a CODEC module 2125, and a JPU 2126. The FuSa fault processing system 22 includes an fault collection module 221, an identification module 222, and an arbitration module 223.

[0032] Here, the input terminal of the data collection module 211 is connected to the output terminal of the vehicle-mounted camera. The processing module 212 is connected between the output terminal of the data collection module 211 and the input terminal of the intelligent driving algorithm module 213. The interface 214 is connected to the input terminal of the abnormality collection module 221, and the output terminal of the abnormality collection module 221 is connected to the input terminal of the identification module 222. The output terminal of the identification module 222 is connected to the input terminal of the arbitration module 223, and the output terminal of the arbitration module 223 is connected to the interface 214.

[0033] The image processing process will now be described by way of example with reference to FIG.

[0034] As shown in FIG. 2, first, a user inputs a module connection setting operation to the processing module 212 in the VPS 21, and in response to this module connection setting operation, the processing module 212 establishes a video path based on the MIPI module 2121, the CIM 2122, the ISP module 2123, the PYM module 2124, the CODEC module 2125, the JPU 2126, etc. (In some examples, the video path may include some image processing modules in the processing module 212. For example, the video path may include the MIPI module 2121, the ISP module 2123, the PYM module 2124, and the JPU 2126 connected in sequence).

[0035] Then, the user inputs an operation mode setting operation to the processing module 212 in the VSP21, and the processing module 212 determines the operation mode setting information in response to this operation mode setting operation, and determines its own processing module for intercepting and processing data from each path module of the video path based on the operation mode setting information.

[0036] Next, the data collection module 211 acquires original image data (raw images) from the output terminal of the vehicle-mounted camera, and outputs the acquired raw images to the input terminal of the processing module 212.

[0037] Next, each path module in the video path of the processing module 212 sequentially processes the input raw image to obtain processed image data, and outputs the processed raw image to the intelligent driving algorithm module 213.

[0038] Finally, the intelligent driving algorithm module 213 generates intelligent driving data according to the processed raw image, and outputs the intelligent driving data to the intelligent driving system to realize intelligent driving for the vehicle.

[0039] Take for example a FuSa error occurring in CIM 2122 during the process of each path module in the video path of processing module 212 sequentially processing raw images. The FuSa error handling process is as follows:

[0040] First, the CIM 2122 inputs the anomaly information to the interface 214. The interface 214 receives the anomaly information and reports the anomaly information to the anomaly collection module 221.

[0041] Next, the anomaly collection module 221 receives the anomaly information and outputs the anomaly information to the identification module 222 .

[0042] Next, the identification module 222 receives the abnormality information, searches the abnormality processing correspondence relationship (e.g., an abnormality processing policy table) stored in the database based on the abnormality information, determines a processing policy to resolve this FuSa abnormality, and outputs this processing policy and the CIM2122 in which the FuSa abnormality occurred (e.g., the name of the image processing module or an identifier representing the image processing module) to the arbitration module 223.

[0043] Finally, the arbitration module 223 determines a target processing module from each path module in the video path based on the operating mode setting information, the CIM 2122 and the processing policy, generates a processing operation command corresponding to the target processing module, and transmits it to the target processing module.

[0044] In the embodiment of the present disclosure, the target processing module is determined from each path module in the video path based on the operation mode setting information, the abnormal path module, and the processing policy, and the abnormal path module, the coupling relationship between the abnormal path module and other path modules, and the abnormality in the actual use scenario of image processing are comprehensively considered. Therefore, when the determined target processing module is processed, the FuSa abnormality can be effectively resolved, and the occurrence of problems such as the entire VPS abnormality and crash can be avoided, and the safety of intelligent operation can be improved.

[0045] [Exemplary Method] 3 is a flowchart of an image processing method using a system on a chip according to an exemplary embodiment of the present disclosure. As shown in FIG. 3, the image processing method using the system on a chip may include the following steps 301 to 304.

[0046] In step 301, abnormality information occurring during the process of processing a first image data frame by a plurality of image processing modules in the system-on-chip and path setting information corresponding to the plurality of image processing modules are determined.

[0047] For example, the system on chip may be an intelligent driving chip, such as the intelligent driving chip 20 shown in FIG.

[0048] Illustratively, the plurality of image processing modules may include all or some of the image processing modules in the VPS system and may be determined based on the connection setting information in the path setting information. In some examples, as shown in FIG. 2, the plurality of image processing modules may include at least one image processing module among image processing modules such as the MIPI module 2121, the CIM 2122, the ISP module 2123, the PYM module 2124, the CODEC module 2125, and the JPU 2126. If the connection setting information indicates that the MIPI module 2121, the ISP module 2123, the PYM module 2124, and the JPU 2126 are sequentially connected in series, the plurality of image processing modules may be determined to include the MIPI module 2121, the ISP module 2123, the PYM module 2124, and the JPU 2126. In other examples, because the plurality of image processing modules are modules that process a first image data frame, the plurality of image processing modules may be determined as a plurality of path modules in the video path.

[0049] Illustratively, the first image data frame can be the first frame image data processed by the image processing modules or the image data being processed at the current time. In some examples, as shown in FIG. 2, the first image data frame can be a raw image output by the in-vehicle camera to the data collection module 211 at the current time and transmitted by the data collection module 211 to the processing module 212.

[0050] For example, the anomaly information may be any information that explains the occurrence of the FuSa anomaly. In some examples, the anomaly information may include specific information about the anomaly and / or an ID number of the anomaly (the ID number can determine the type of the FuSa anomaly), but the embodiments of the present disclosure are not limited thereto. The specific information about the anomaly may include specific information such as the name of the image processing module or sub-module in which the FuSa anomaly occurred and the type of the FuSa anomaly.

[0051] For example, the path setting information may be information determined based on a path setting operation input by a user. The path setting information may include connection mode setting information and operation mode setting information. In some examples, the connection mode setting information and the operation mode setting information may be realized by the same setting operation or different setting operations, and the embodiments of the present disclosure are not limited thereto.

[0052] For example, as shown in FIG. 2, the intelligent driving chip 20 can first obtain a path setting operation input by a user and determine path setting information based on the path setting operation input by the user. Based on the connection setting information in the path setting information, the processing module 212 determines a plurality of path modules included in the video path. Based on the operation mode setting information in the path setting information, each path module determines its own processing module for intercepting and processing data. The determined path modules then process raw images input from the vehicle-mounted camera. During the process of the plurality of path modules processing the raw images, if an FuSa anomaly occurs in at least one path module, the at least one path module with the FuSa anomaly generates anomaly information corresponding to the path module and reports it to the anomaly collection module 221 via the interface 214. Finally, the anomaly collection module 221 obtains the anomaly information and outputs it to the identification module 222.

[0053] In step 302, the abnormal module where the abnormal information occurs and the initial processing policy for handling the abnormal information are determined.

[0054] For example, the initial processing policy may be a conservative processing policy preset for different anomaly types and updated based on the effectiveness of the policy. In some examples, the initial processing policy may include processing policies of different levels. For example, the initial processing policy may include, but is not limited to, sequentially increasing levels of frame dropping, module reset, upstream module reset, downstream module frame dropping, and system reset.

[0055] For example, as shown in FIG. 2 , the identification module 222 can obtain anomaly information from the output terminal of the anomaly collection module 221, determine an abnormal image processing module and an initial processing policy corresponding to the anomaly information based on the anomaly information and a plurality of preset processing policies, and output the abnormal image processing module and the initial processing policy to the arbitration module 223.

[0056] In step 303, a target processing module is determined from the plurality of image processing modules based on the abnormal module, the initial processing policy, and the path setting information.

[0057] For example, the target processing module may be at least one of a plurality of image processing modules. In some examples, the target processing module may include an anomaly module or may not include an anomaly module. In other examples, the target processing module may include at least a self-processing module, although the embodiments of the present disclosure are not limited thereto.

[0058] 2, the arbitration module 223 can receive the abnormal image processing module, the initial processing policy, and the operation mode setting information in the path setting information. Based on the abnormal image processing module, the initial processing policy, and the operation mode setting information in the path setting information, at least one path module from the multiple path modules is determined as the target processing module according to a predetermined image processing module determination rule.

[0059] In step 304, the target processing module is processed based on the initial processing policy.

[0060] For example, a target processed module can be obtained, a corresponding processing operation instruction can be generated based on the target processed module and an initial processing policy, the internal logic in the target processed module can be processed based on the processing operation instruction, and a processed target processed module can be obtained.

[0061] 2, after determining the target processing module, the arbitration module 223 can generate an initial processing policy and a processing operation command corresponding to the target processing module, and transmit the processing operation command to the corresponding target processing module. The target processing module performs a frame drop or reset process in response to the processing operation command, and obtains a target processing module that has been subjected to the frame drop or reset process. After obtaining the processed target processing module, the processed processing module processes the second image data frame.

[0062] In some examples, different initial processing policies and / or different target processed modules may correspond to different processing operation instructions. For example, if the initial processing policy is frame drop and the target processed module is the ISP module 2123, the processing operation instruction for the ISP module 2123 may be a frame drop instruction for the ISP module 2123. If the initial processing policy is module reset and the target processed modules are the ISP module 2123 and the MIPI module 2121, an upstream path module of the ISP module 2123, the processing operation instructions for the ISP module 2123 and the MIPI module 2121 may be a reset instruction for the ISP module 2123 and a reset instruction for the MIPI module 2121, respectively.

[0063] Step 305 processes the second image data frame with the processed target processing object module.

[0064] 2, the second image data frame may be a raw image after the first image data frame, and this raw image may be output by the on-board camera to the data collection module 211, which may then output it to the processing module 212. In some examples, the acquisition times of the second image data frame and the first image data frame may be consecutive or non-consecutive, and the embodiments of the present disclosure are not limited thereto.

[0065] In an embodiment of the present disclosure, abnormality information generated during processing of a first image data frame by multiple image processing modules in a system-on-chip and path setting information corresponding to the multiple image processing modules are determined, and an abnormal module and an initial processing policy are determined based on the abnormality information. Finally, a target processing module is determined from the multiple image processing modules based on the abnormal module, the initial policy, and the path setting information. That is, in determining the target processing module, the abnormal path module, the coupling relationship between the abnormal path module and other path modules, and abnormalities in actual image processing scenarios are comprehensively considered. Therefore, processing the target processing module determined based on the initial processing policy can effectively resolve the FuSa abnormality, avoid problems such as an abnormality or crash of the entire VPS, and improve the safety of intelligent operation.

[0066] As shown in FIG. 4, based on the embodiment shown in FIG. 3 above, step 302 may include the following steps 3021 and 3022.

[0067] In step 3021, the anomaly information is analyzed to determine an anomaly module, a module identifier of the anomaly module, and / or an anomaly type of the anomaly module.

[0068] For example, the module identifier of the abnormal module may represent or represent a character or string of characters of the abnormal module. In some examples, the module identifier of the abnormal module may be a string of characters composed of numbers and letters, and the embodiments of the present disclosure do not specifically limit the composition of the string. For example, the module identifier of the abnormal module may be S1 or S2.

[0069] For example, the anomaly type may be the name of the anomaly that has occurred, and may include anomalies such as a registration task anomaly, a line anomaly, a parity check anomaly, a cyclic redundancy check anomaly, a size anomaly, an error correction code anomaly, a potential anomaly, and a non-fatal anomaly. In some examples, the anomaly type may be determined according to the main object or processing operation in which the anomaly has occurred. For example, if the main object in which the anomaly has occurred is a line, the anomaly type may be determined to be a line anomaly. If the processing operation in which the anomaly has occurred is a parity check, the anomaly type may be determined to be a parity check anomaly.

[0070] For example, as shown in FIG. 2, the identification module 222 may receive the anomaly information and analyze the anomaly information to obtain the name of the anomaly module, the identifier of the anomaly module, and the anomaly type of the anomaly module.

[0071] In step 3022, an abnormality processing correspondence relationship is searched based on the abnormal module, module identifier and / or abnormality type, and a processing policy corresponding to the abnormal module, module identifier and / or abnormality type is determined as the initial processing policy.

[0072] For example, the anomaly processing correspondence relationship can be expressed as a plurality of processing policies corresponding to a plurality of types of anomalies, where one type of anomaly can correspond to one processing policy.

[0073] In some examples, the anomaly processing correspondence relationship may include a correspondence relationship between a module identifier and a processing policy, a correspondence relationship between a module name and a processing policy, and a correspondence relationship between an anomaly type and a processing policy. For example, the anomaly processing correspondence relationship is as shown in Table 1 below.

[0074] [Table 1]

[0075] For example, take the case where the abnormality type is a parity check abnormality, the module identifier is S2, and the abnormal module is a CIM. By searching Table 1, the abnormality type is a parity check code abnormality, the module identifier is S, and the abnormal module is a CIM, and the processing policy "reset module" corresponding to the abnormality type is a parity check code abnormality, the module identifier is S2, and the abnormal module is a CIM can be determined as the initial processing policy.

[0076] In an embodiment of the present disclosure, the abnormality information is analyzed to determine the abnormal module, the module identifier of the abnormal module, and the abnormality type of the abnormal module, and an abnormality handling correspondence relationship is searched based on the abnormal module, the module identifier of the abnormal module, and the abnormality type of the abnormal module, and a handling policy corresponding to the abnormal module, the module identifier of the abnormal module, and the abnormality type of the abnormal module is determined as the initial handling policy. In this way, if the abnormality handling correspondence relationship can accurately represent the actual FuSa abnormal situation and the handling policy provided for the FuSa abnormality is effective, an effective initial handling policy can be accurately determined.

[0077] As shown in FIG. 5, in the embodiment shown in FIG. 3 above, step 302 may include step 3023 and step 3024.

[0078] In step 3023, the anomaly information is analyzed to determine the anomaly type of the anomaly module.

[0079] Illustratively, the implementation manner of step 3023 can refer to step 3021, and the description thereof will be omitted in the embodiments of the present disclosure.

[0080] In step 3024, an abnormality processing correspondence relationship is searched based on the abnormality type of the abnormal module, and a processing policy corresponding to the abnormality type is determined as the initial processing policy.

[0081] For example, if the anomaly type is an error correction code anomaly, then by searching Table 1, the processing policy corresponding to the error correction code anomaly in Table 1, "reset + frame drop", can be determined as the initial processing policy.

[0082] In some examples, the abnormality information is analyzed to determine the abnormal module, and based on the abnormal module, the abnormality processing correspondence relationship is searched to determine the processing policy corresponding to the abnormal module as the initial processing policy. Take the example where the abnormal module is a CIM. Since the name of the abnormal module is CIM, Table 1 can be searched to determine "reset module", which is the processing policy corresponding to CIM in Table 1, as the initial processing policy.

[0083] In some other examples, the abnormality information is analyzed to determine the module identifier of the abnormal module, and the abnormality processing correspondence relationship is searched based on the module identifier of the abnormal module, and the processing policy corresponding to the module identifier is determined as the initial processing policy. Take the example where the module identifier is S2. Since the module identifier is S2, Table 1 can be searched, and the processing policy "reset module", which is the processing policy corresponding to S2 in Table 1, can be determined as the initial processing policy.

[0084] In some other examples, the anomaly information can be analyzed to determine two elements among the anomaly module, the module identifier of the anomaly module, and the anomaly type of the anomaly module, and the anomaly processing correspondence relationship can be searched for based on these two elements, and the processing policy corresponding to these two elements can be determined as the initial processing policy. Taking the two elements as an example, the anomaly information can be analyzed to determine the anomaly module and the anomaly type of the anomaly module, and the anomaly processing correspondence relationship can be searched for based on the anomaly module and the anomaly type of the anomaly module, and the processing policy corresponding to the anomaly module and the anomaly type of the anomaly module can be determined as the initial processing policy.

[0085] In an embodiment of the present disclosure, the anomaly type of the abnormal module is determined by analyzing the anomaly information, and the anomaly processing correspondence relationship is searched based on the anomaly type of the abnormal module, and the processing policy corresponding to the anomaly type of the abnormal module is determined as the initial processing policy. In this way, if the anomaly processing correspondence relationship can accurately represent the actual FuSa abnormal situation and the processing policy provided for the FuSa abnormality is effective, the effective initial processing policy can be accurately determined.

[0086] In some embodiments of the present disclosure, the path setting information may include first mode setting information and second mode setting information.

[0087] As shown in FIG. 6, based on the embodiment shown in FIG. 3 above, step 303 may include the following steps 3031 and 3032.

[0088] In step 3031, in response to the initial processing policy being one of frame drop, module reset, and partial reset and partial frame drop, at least one pass module is determined from the plurality of image processing modules based on the first mode setting information.

[0089] Exemplarily, the first mode setting information may be for setting a connection mode of each image processing module. The first mode setting information may be operation mode setting information in the embodiment shown in FIG. 1 or 2. In some examples, the first mode setting information may include different identifiers corresponding to different connection modes (video paths). For example, the first mode setting information may include an identifier CON1 corresponding to the first connection mode or an identifier CON2 corresponding to the second connection mode.

[0090] 2, the first connection mode may correspond to a video path formed by sequentially connecting the MIPI module 2121, the ISP module 2123, the PYM module 2124, and the JPU 2126. The second connection mode may correspond to a video path formed by sequentially connecting the MIPI module 2121, the CIM 2122, the ISP module 2123, the PYM module 2124, and the JPU 2126.

[0091] For example, if the initial policy is one of frame drop, module reset, and partial reset and partial frame drop, a pass module for processing the first image data frame from multiple image processing modules and the connection relationship between each pass module can be determined based on the first mode setting information.

[0092] 2, when the arbitration module 223 receives an initial policy and first mode setting information and determines that the initial policy is one of frame drop, module reset, and partial reset and partial frame drop, it analyzes the first mode setting information to determine a path module for processing the set first image data frame and the connection relationship between the path modules. For example, when the first mode setting information includes an identifier CON1, it determines that at least one path module includes a MIPI module 2121, an ISP module 2123, a PYM module 2124, and a JPU 2126, and that the MIPI module 2121, the ISP module 2123, the PYM module 2124, and the JPU 2126 are sequentially connected.

[0093] In step 3032, a target processing module is determined from at least one path module based on the abnormal module and the second mode setting information.

[0094] Here, at least one path module includes an abnormal module.

[0095] Exemplarily, the second setting information may be for setting an operation mode of each image processing module. The second mode setting information may be the operation mode setting information in the embodiment shown in FIG. 1 or 2. In some examples, the second setting information may include identifiers for representing different usage scenes of the client. For example, the second setting information may include an identifier "Front" corresponding to a forward-looking scene, an identifier "Back" corresponding to a rearward-looking scene, an identifier "Round" corresponding to a look-around scene, and an identifier "Side" corresponding to a side-viewing scene.

[0096] 2, the arbitration module 223 can analyze the second mode setting information to obtain the user's usage scene, and determine a target processing module from at least one path module based on the user's usage scene and a predetermined image processing module determination rule. For example, when the second setting information is analyzed to obtain an identifier Front, the arbitration module 223 can determine the user's usage scene as a forward viewing scene, determine the operating characteristics of each path module in the forward viewing scene, and determine the target processing module based on the operating characteristics of each path module in the forward viewing scene and the predetermined image processing module determination rule.

[0097] In an embodiment of the present disclosure, when the initial policy is one of frame drop, module reset, and partial reset and partial frame drop, at least one pass module is determined from the plurality of image processing modules according to the first mode setting information, so that the target processing module for processing the first image data frame from the VPS can be accurately determined.

[0098] As shown in FIG. 7, based on the embodiment shown in FIG. 6 above, step 3032 may include the following steps 701 and 702.

[0099] In step 701, based on the second mode setting information and at least one path module, a containment relationship between a plurality of path modules downstream of the abnormal module and a self-processing module for intercepting and processing data is determined.

[0100] Exemplarily, the plurality of path modules downstream of the abnormal module refer to the plurality of path modules connected after the abnormal module in the video path. In some examples, the plurality of path modules downstream of the abnormal module can be determined based on the abnormal module and the first mode setting information. For example, as shown in FIG. 2, if the video path corresponding to the first mode setting information is formed by sequentially connecting the MIPI module 2121, the ISP module 2123, the PYM module 2124, and the JPU 2126, and the abnormal module is the MIPI module 2121, the ISP module 2123, the PYM module 2124, and the JPU 2126 can be determined to be the plurality of path modules downstream of the abnormal module.

[0101] Exemplarily, the self-processing module may be an image processing module configured to implement data interception and processing functions in an image processing module having data interception and processing functions. In some examples, the self-processing module may be an image processing module that stores the processed image data in a synchronous dynamic random access memory (Double Data Rate (SDRAM) (DDR)) during the process of processing the image data. For example, as shown in FIG. 2, the self-processing module may be an ISP module 2123.

[0102] Illustratively, the containment relationship can be containment or non-containment, and the embodiments of the present disclosure are not specifically limited thereto.

[0103] In some examples, when the inclusion relationship is inclusive, the multiple path modules downstream of the abnormal module include self-processing modules for intercepting and processing data. For example, take a video path including a MIPI module 2121, an ISP module 2123, a PYM module 2124, and a JPU 2126 connected in sequence, the abnormal module is the MIPI module 2121, and the multiple path modules downstream of the abnormal module include the ISP module 2123, the PYM module 2124, and the JPU 2126. If it is determined based on the second mode setting information that the ISP module 2123 having the data intercepting and processing function is configured to realize the data intercepting and processing function, that is, if the ISP module 2123 is a self-processing module, it is determined that the ISP module 2123, the PYM module 2124, and the JPU 2126 of the multiple path modules downstream of the abnormal module include self-processing modules for intercepting and processing data.

[0104] In some other examples, when the inclusion relationship is non-inclusive, the multiple path modules downstream of the abnormal module do not include a self-processing module for intercepting and processing data. For example, take the example where the video path includes a MIPI module 2121, an ISP module 2123, a PYM module 2124, and a JPU 2126 connected in sequence, the abnormal module is the PYM module 2124, and the multiple path modules downstream of the abnormal module include the JPU 2126. If it is determined based on the second mode setting information that the ISP module 2123 having the data intercepting and processing function is configured to realize the data intercepting and processing function, that is, the ISP module 2123 is a self-processing module, it is determined that the multiple path modules downstream of the abnormal module, the JPU 2126, do not include a self-processing module for intercepting and processing data.

[0105] For example, the second mode setting information can be analyzed, the user's usage scenario can be determined based on the analysis result, and a self-processing module (operating characteristics corresponding to each path module) for intercepting and processing data corresponding to the user's usage scenario can be determined, and then it can be determined whether multiple path modules downstream of the abnormal module in the video path include a self-processing module for intercepting and processing data corresponding to the usage scenario.

[0106] In some examples, as shown in Figure 2, the arbitration module 223 analyzes the second mode setting information, and when the second mode setting information includes an identifier Front and the identifier Front is obtained through analysis, the arbitration module 223 can determine that the user's usage scene is a forward-looking scene, and then determine whether the path module downstream of the abnormal module includes an ISP module 2123, which is a self-processing module corresponding to the forward-looking scene.

[0107] In step 702, a target processing module is determined from at least one path module based on the inclusion relationship.

[0108] For example, if the determination result is that the path module downstream of the abnormal module includes a self-processing module for intercepting and processing data, a target processing module needs to be determined from at least one path module based on the initial processing policy and the self-processing module; if the determination result is that the path module downstream of the abnormal module does not include a self-processing module for intercepting and processing data, a target processing module needs to be determined based on at least one path module.

[0109] In some examples, in response to the initial processing policy being frame drop and multiple path modules downstream of the abnormal module including a self-processing module, the self-processing module can be determined as the target processing target module, and in response to the initial processing policy being frame drop and multiple path modules downstream of the abnormal module not including a self-processing module, at least one path module can be determined as the target processing target module.

[0110] For example, as shown in FIG. 2, if the abnormal module is the MIPI module 2121, the initial processing policy is frame drop, the self-processing module is the ISP module 2123, and the video path includes the MIPI module 2121, the ISP module 2123, the PYM module 2124, and the JPU 2126, that is, if the path module downstream of the abnormal module includes the ISP module 2123, the arbitration module 223 can determine the ISP module 2123 as the target processing module. If the abnormal module is the PYM module 2124, the initial policy is frame drop, the self-processing module is the ISP module 2123, and the video path includes the MIPI module 2121, the ISP module 2123, the PYM module 2124, and the JPU 2126, i.e., if the path modules downstream of the abnormal module do not include the ISP module 2123, the arbitration module 223 can determine the MIPI module 2121, the ISP module 2123, the PYM module 2124, and the JPU 2126 (all path modules in the video path) as the target processing modules.

[0111] In an embodiment of the present disclosure, the initial processing policy is frame drop, and multiple path modules downstream of the abnormal module include a self-processing module, thereby determining the self-processing module as the target processing module. In this way, the FuSa abnormality occurring in the abnormal module can be quickly and easily resolved based on this self-processing module. In this way, the initial processing policy is frame drop, and multiple path modules downstream of the abnormal module do not include a self-processing module, thereby determining at least one path module as the target processing module. In this way, the FuSa abnormality occurring in the abnormal module can be effectively resolved by the cooperation of at least one path module.

[0112] In some examples, in response to the initial processing policy being to reset the module and the plurality of path modules downstream of the abnormal module including the self-processing module, the self-processing module and the path module upstream of the self-processing module are determined as target processing target modules, and in response to the initial processing policy being to reset the module and the plurality of path modules downstream of the abnormal module not including the self-processing module, at least one path module is determined as the target processing target module.

[0113] For example, as shown in FIG. 2, if the abnormal module is the MIPI module 2121, the initial processing policy is module reset, the self-processing module is the ISP module 2123, and the video path includes the MIPI module 2121, the ISP module 2123, the PYM module 2124, and the JPU 2126, that is, if the path module downstream of the abnormal module includes the ISP module 2123, the arbitration module 223 can determine the ISP module 2123 and the MIPI module 2121 upstream of the ISP module 2123 as the target processing target module. If the abnormal module is the PYM module 2124, the initial policy is module reset, the self-processing module is the ISP module 2123, and the video path includes the MIPI module 2121, the ISP module 2123, the PYM module 2124, and the JPU 2126, i.e., if the path modules downstream of the abnormal module do not include the ISP module 2123, the arbitration module 223 can determine the MIPI module 2121, the ISP module 2123, the PYM module 2124, and the JPU 2126 as the target processing modules.

[0114] In an embodiment of the present disclosure, the initial processing policy is to reset the module, and multiple path modules downstream of the abnormal module include a self-processing module, thereby determining the self-processing module and the path module upstream of the self-processing module as target processing modules. In this manner, the FuSa abnormality occurring in the abnormal module can be quickly and easily resolved based on the self-processing module and the path module upstream of the self-processing module. In this manner, the initial processing policy is to reset the module, and multiple path modules downstream of the abnormal module do not include the self-processing module, thereby determining at least one path module as the target processing module. In this manner, the FuSa abnormality occurring in the abnormal module can be effectively resolved by the cooperation of at least one path module.

[0115] In some examples, in response to the initial processing policy being a partial reset and partial frame drop, and multiple path modules downstream of the abnormal module including a self-processing module, when resetting the path module, the self-processing module and the path module upstream of the self-processing module are determined as target processing target modules, and when processing the path module for frame drop, the path module downstream of the self-processing module is determined as target processing target module.

[0116] For example, partial reset and partial frame drop refers to processing the first image data frame twice, one time to reset and the other time to drop frames. In some examples, some path modules in the video path may be reset, and other path modules in the video path may be frame dropped. For example, as shown in FIG. 2, if the video path includes a MIPI module 2121, an ISP module 2123, a PYM module 2124, and a JPU 2126, the MIPI module 2121 and the ISP module 2123 may be reset, and the PYM module 2124 and the JPU 2126 may be frame dropped.

[0117] 2, if the abnormal module is the MIPI module 2121, the initial processing policy is partial reset and partial frame drop, the self-processing module is the ISP module 2123, and the video path includes the MIPI module 2121, the ISP module 2123, the PYM module 2124, and the JPU 2126, i.e., if the path module downstream of the abnormal module includes the ISP module 2123, when the arbitration module 223 resets the path module, it determines the ISP module 2123 and the MIPI module 2121 upstream of the ISP module 2123 as the target modules to be processed. When the arbitration module 223 processes the path module for frame drop, it determines the PYM module 2124 and the JPU 2126 downstream of the ISP module 2123 as the target modules to be processed.

[0118] In an embodiment of the present disclosure, the initial processing policy is partial reset and partial frame drop, and multiple path modules downstream of an abnormal module include a self-processing module. Therefore, when a path module is reset, the self-processing module and the path module upstream of the self-processing module are determined as the target modules to be processed, and when a path module is frame-dropped, the path module downstream of the self-processing module is determined as the target module to be processed. In this way, by resetting the self-processing module and the path module upstream of the self-processing module and dropping frames for the path module downstream of the self-processing module, the FuSa abnormality occurring in the abnormal module can be quickly and easily resolved.

[0119] In an embodiment of the present disclosure, an inclusion relationship between multiple path modules downstream of the abnormal module and a self-processing module for intercepting and processing data is determined based on the second mode setting information and at least one path module, and a target processing module is determined from the at least one path module based on the inclusion relationship, thereby accurately determining a target processing module from the at least one path module that can quickly and easily resolve the FuSa abnormality based on the second mode setting information.

[0120] As shown in FIG. 8, based on the embodiment shown in FIG. 3 above, step 303 may include the following step 3033:

[0121] In step 3033, in response to the initial processing policy being a system reset, a plurality of image processing modules are determined as target processing target modules.

[0122] For example, as shown in FIG. 2, if the identification module 222 determines that the initial processing policy is a system reset, it can report a system reset request to the in-vehicle microcontroller unit (MCU), and the in-vehicle MCU can reset all image processing modules in the VSP 21.

[0123] In an embodiment of the present disclosure, when the initial processing policy is to reset the system, multiple image processing modules are determined as target processing target modules, thereby realizing resetting the VPS in the in-vehicle system.

[0124] As shown in FIG. 9, in addition to the embodiment shown in FIG. 3, the image processing method using a system-on-chip further includes the following steps 306 to 308.

[0125] In step 306, a first processing result is determined based on the processed target processing object module.

[0126] For example, the first processing result is a result of whether the processing for the target processing module was successful or not. The first processing result may include a processing result string representing whether the processing was successful or not. In some examples, the first processing result may include "YES" to represent the processing was successful or "NO" to represent the processing was not successful. In other examples, the first processing result may include "00" to represent the processing was successful or "11" to represent the processing was not successful. The embodiments of the present disclosure are not limited thereto.

[0127] Illustratively, the target processing module may receive a processing operation instruction, process its internal logic in response to the processing operation instruction, and then generate a first processing result based on the state parameters of the target processing module or the processed image data.

[0128] 2 , when the target processing module is the ISP module 2123 and the processing operation is a 5-frame drop operation, the ISP module 2123, in response to the processing operation, discards 5 frames of data in the image data output by the ISP module 2123 after processing the first image data frame, to obtain image data after frame dropping. The ISP module 2123 generates a first processing result to indicate whether the frame dropping has been successful or unsuccessful based on the image data after frame dropping, and reports the first processing result to the arbitration module 223.

[0129] For example, if the ISP module 2123 determines that the image data after the frame drop is the same in size as the image data before the frame drop, it determines that the frame drop has failed, generates a "NO" to indicate that the processing has failed, and reports this "NO" as the first processing result to the arbitration module 223. If the ISP module 2123 determines that the image data after the frame drop is five frames smaller than the image data before the frame drop, it determines that the frame drop has been successful, generates a "YES" to indicate that the processing has been successful, and reports this "YES" as the first processing result to the arbitration module 223.

[0130] In step 307, in response to the first processing result indicating that the processing for the target processing module was successful, it is determined whether abnormal information occurred during the processing of the second image data frame by the processed target processing module.

[0131] For example, as shown in FIG. 2, if the arbitration module 223 analyzes the received first processing result and determines that the first processing result indicates that the frame drop processing for the ISP module 2123 has been successful, it determines whether the same abnormal module and the same initial processing policy have been received again within a predetermined period of time, that is, whether abnormal information has occurred during the process of the processed target processing module processing the second image data frame.

[0132] In step 308, in response to abnormal information occurring during the processing of the second image data frame by the processed target processing module, it is determined that the initial processing policy is invalid, the initial processing policy is upgraded, a new target processing module corresponding to the upgraded processing policy is determined, and the new target processing module is processed according to the upgraded processing policy to obtain the processed new target processing module.

[0133] 2, if the arbitration module 223 continues to receive the same abnormal module and the same initial processing policy, it determines that the initial processing policy is invalid and the FuSa abnormality occurring in the abnormal module is not resolved. Then, the arbitration module 223 upgrades the processing policy, determines a new target processing target module corresponding to the upgraded processing policy based on the abnormal module and the second mode setting information, processes the new target processing target module according to the upgraded processing policy, and obtains the processed new target processing target module.

[0134] In some examples, if the initial processing policy is 3-frame drop data, the upgraded processing policy may be 7-frame drop data. In this case, the new target processing module corresponding to the upgraded processing policy may be the same as the target processing module corresponding to the initial processing policy. For example, the new target processing module corresponding to the upgraded processing policy and the target processing module corresponding to the initial processing policy may both be the ISP module 2123.

[0135] In some other examples, when the initial processing policy is to drop 10 frames of data, the upgraded processing policy may reset the module. In this case, the new target processing module corresponding to the upgraded processing policy is different from the target processing module corresponding to the initial processing policy. For example, the target processing module corresponding to the initial processing policy may be the ISP module 2123, and the new target processing modules corresponding to the upgraded processing policy may be the MIPI module 2121 and the ISP module 2123.

[0136] For example, an implementation form of determining a new target processing module corresponding to an upgraded processing policy based on an abnormal module and second mode setting information may be similar to an implementation form of determining a target processing module corresponding to an initial processing policy based on an abnormal module and second mode setting information, and reference may be made to the embodiments shown in Figures 6 and 7, and the description thereof will be omitted in the embodiments of the present disclosure.

[0137] For example, an implementation form of processing a new target processing target module according to an upgraded processing policy to obtain a processed new target processing target module is similar to an implementation form of processing a target processing target module according to an initial processing policy to obtain a processed target processing target module, and reference can be made to the embodiment shown in Figure 3, and this description will be omitted in the embodiments of the present disclosure.

[0138] In an embodiment of the present disclosure, if processing of the target processing module is successful, it is determined whether abnormality information should be generated when the processed target processing module processes the second image data frame. If abnormality information is generated when the processed target processing module processes the second image data frame, the invalidation of the initial processing policy is determined. In this way, the validity of the initial processing policy can be accurately determined. Then, the initial processing policy is upgraded, a new target processing module is determined using the upgraded processing policy, and the new target processing module is processed using the upgraded processing policy to obtain the new processed target processing module. In this way, upgrading the initial processing policy and processing the new target processing module using the upgraded processing policy can more effectively resolve the FuSa abnormality than using the initial processing policy.

[0139] As shown in FIG. 10, in addition to the embodiment shown in FIG. 9 above, the image processing method using a system-on-chip further includes step 309 and step 310.

[0140] In step 309, a second processing result is determined based on the processed new target processing module.

[0141] For example, the second processing result is a result of whether the processing for the new target processing module is successful or not. The second processing result may be similar to the first processing result and include a processing result indicating whether the processing is successful or unsuccessful, which will not be specifically described in the embodiments of the present disclosure.

[0142] Illustratively, the implementation of step 309 is similar to that of step 306, and the description thereof is omitted here.

[0143] In step 310, in response to the second processing result indicating that the processing for the processed new target processing module is successful and no abnormal information occurs during the processing of the third image data frame by the processed new target processing module, it is determined that the upgraded processing policy is valid and the abnormal processing correspondence relationship is updated according to the upgraded processing policy.

[0144] 2, the third image data frame may be a raw image output from the camera after the second image data frame to the processing module 212. In some examples, the acquisition period of the third image data frame and the second image data frame may be continuous or discontinuous, and the embodiments of the present disclosure are not limited thereto.

[0145] 2, if the arbitration module 223 determines that the new target processing module has been successfully processed and does not receive the same abnormal module or the same initial processing policy during processing of the third image data frame, the arbitration module 223 determines that the upgraded processing policy is valid. Then, the arbitration module 223 can update the abnormal processing correspondence in the identification module 222 according to the upgraded processing policy.

[0146] In an embodiment of the present disclosure, a second processing result is determined based on the processed new target processing module. If the second processing result indicates that the processing for the processed new target processing module was successful, it is determined whether abnormality information occurs during the processing of the third image data frame by the processed new target processing module. If no abnormality information occurs during the processing of the third image data frame by the processed new target processing module, it is determined that the upgraded processing policy is valid. In this way, the validity of the upgraded processing policy can be accurately determined based on whether abnormality information occurs. Furthermore, the abnormality processing correspondence is updated based on the upgraded processing policy. In this way, the next time the same FuSa abnormality occurs, the target processing module can be processed using the valid processing policy, thereby improving the efficiency of FuSa abnormality processing.

[0147] As shown in FIG. 11, in the embodiment shown in FIG. 3 above, the image processing method using a system on a chip further includes step 311 and step 312.

[0148] In step 311, the number of times that abnormal information occurs in the abnormal module within a predetermined period is determined.

[0149] For example, the predetermined period may be a preset threshold value and may be determined based on the abnormality processing period of the VPS. In some examples, the predetermined period may be less than 10 seconds. For example, the predetermined period may be 5 seconds.

[0150] For example, as shown in FIG. 2, the arbitration module 223 can count the number of times the same abnormal module and the same initial processing policy are received from the identification module 222 within a predetermined period, and determine the number of times the same abnormal module and the same initial processing policy are received as the number of times abnormal information occurs in the abnormal module.

[0151] In step 312, in response to the number of times being equal to or greater than the number threshold, the target processing target module is processed based on a predetermined processing policy to obtain a processed target processing target module.

[0152] Here, the predetermined processing policy is an effective policy for resolving the abnormality information.

[0153] For example, the count threshold may be determined according to the risk level of the FuSa anomaly. The higher the risk level, the smaller the count threshold, and the lower the risk level, the larger the count threshold. In some examples, the count threshold may be any integer greater than or equal to 1. For example, the count threshold may be 10.

[0154] For example, the level of the predetermined processing policy may be higher than the initial processing policy. In some examples, the predetermined processing policy may be a processing policy that can quickly resolve the FuSa anomaly. For example, if the initial processing policy is frame drop, the predetermined processing policy may be module reset. If the initial processing policy is module reset, the predetermined processing policy may be system reset. The embodiments of the present disclosure are not limited thereto.

[0155] For example, as shown in FIG. 2, the arbitration module 223 determines whether the number of times abnormal information occurs in the abnormal module within a specified period is greater than or equal to the count threshold. If the number of times abnormal information occurs in the abnormal module within a specified period is greater than or equal to the count threshold, the arbitration module 223 directly upgrades the initial processing policy to a specified processing policy, determines a target processing target module corresponding to the specified processing policy, processes the target processing target module corresponding to the specified processing policy using the specified processing policy, and obtains the processed target processing target module.

[0156] In an embodiment of the present disclosure, the number of times abnormal information occurs in the abnormal module within a predetermined period is determined, and if the number of times abnormal information occurs in the abnormal module within the predetermined period is equal to or greater than a threshold, the target module to be processed is processed according to a predetermined processing policy to obtain the processed target module to be processed. In this way, when the predetermined processing policy is an effective policy, processing the target module to be processed according to the predetermined processing policy can quickly and effectively resolve the FuSa abnormality, improve the efficiency of FuSa abnormality processing, and enhance the safety of intelligent driving.

[0157] [Example Device] 12 is a schematic diagram of an image processing device based on a system on a chip according to an exemplary embodiment of the present disclosure. As shown in FIG. 12, the image processing device based on a system on a chip 120 includes a first determination module 1201, a second determination module 1202, a third determination module 1203, a first processing module 1204, and a second processing module 1205.

[0158] The first determination module 1201 is for determining abnormality information occurring during the process of processing a current image data frame by a plurality of image processing modules in the system-on-chip and path setting information corresponding to the plurality of image processing modules; The second determination module 1202 is for determining the abnormal module where the abnormal information occurs and the initial processing policy for handling the abnormal information; The third determination module 1203 is for determining a target processing module from among a plurality of image processing modules based on the abnormal module, the initial processing policy, and the path setting information; The first processing module 1204 processes the target processing module based on the initial processing policy; The second processing module 1205 is for processing the next image data frame by the processed target processing module.

[0159] In some embodiments, as shown in FIG. 13, in the embodiment shown in FIG. 13 above, the second determination module 1202 includes an analysis unit 1301 and a search unit 1302.

[0160] The analysis unit 1301 is for analyzing the anomaly information to determine an anomaly module, a module identifier of the anomaly module, and / or an anomaly type of the anomaly module; The search unit 1302 is for searching an abnormality processing correspondence relationship based on an abnormal module, a module identifier, and / or an abnormality type, and determining a processing policy corresponding to the abnormal module, a module identifier, and / or an abnormality type as an initial processing policy.

[0161] In some embodiments, as shown in FIG. 14, based on the embodiment shown in FIG. 12 above, the third determination module 1203 includes a first determination unit 1401 and a second determination unit 1402.

[0162] the first determining unit 1401 is for determining at least one pass module from the plurality of image processing modules based on the first mode setting information in response to the initial processing policy being one of frame drop, module reset, and partial reset and partial frame drop; The second determination unit 1402 is for determining a target processing module from at least one path module according to the abnormal module and the second mode setting information; Here, at least one path module includes an abnormal module, the first mode setting information is for setting the connection method of each image processing module, the second setting information is for setting the operating mode of each image processing module, and the path setting information includes the first mode setting information and the second mode setting information.

[0163] In some embodiments, as shown in FIG. 15, based on the embodiment shown in FIG. 14 above, the second determination unit 1402 includes a first determination subunit 1501 and a second determination subunit 1502.

[0164] the first determining subunit 1501 is for determining, according to the second mode setting information and at least one path module, a containment relationship between a plurality of path modules downstream of the abnormal module and a self-processing module for intercepting and processing data; The second determining subunit 1502 is for determining a target processing module from at least one path module based on the inclusion relationship.

[0165] In some embodiments, the second determination subunit 1502 specifically determines the self-processing module as the target processing target module in response to the initial processing policy being frame drop and the plurality of path modules downstream of the abnormal module including the self-processing module, and determines at least one path module as the target processing target module in response to the initial processing policy being frame drop and the plurality of path modules downstream of the abnormal module not including the self-processing module.

[0166] In some embodiments, the second determination subunit 1502 specifically determines the self-processing module and the path module upstream of the self-processing module as target processing target modules in response to the initial processing policy being module reset and the multiple path modules downstream of the abnormal module including the self-processing module, and determines at least one path module as the target processing target module in response to the initial processing policy being module reset and the multiple path modules downstream of the abnormal module not including the self-processing module.

[0167] In some embodiments, the second determination subunit 1502 specifically determines the self-processing module and the upstream path module of the self-processing module as the target processing target module when resetting the path module in response to the initial processing policy being a partial reset and a partial frame drop, and the plurality of path modules downstream of the abnormal module including a self-processing module, and determines the downstream path module of the self-processing module as the target processing target module when processing the path module with frame drop.

[0168] In some embodiments, the second determining subunit 1502 specifically determines the multiple image processing modules as target processing modules in response to the initial processing policy being a system reset.

[0169] In some embodiments, as shown in FIG. 16, in addition to the embodiment shown in FIG. 12 above, the system-on-chip image processing device 120 further includes a fourth determination module 1206, a fifth determination module 1207 and a third processing module 1208.

[0170] The fourth determination module 1206 is for determining the first processing result based on the processed target processing module; The fifth determination module 1207 is for determining whether abnormal information occurs during the process of the processed target processing module processing the second image data frame in response to the first processing result indicating that the processing for the target processing module is successful; The third processing module 1208 is for determining that the initial processing policy is invalid in response to abnormal information occurring during the process of the processed target processing module processing the second image data frame, upgrading the initial processing policy, determining a new target processing module corresponding to the upgraded processing policy, and processing the new target processing module according to the upgraded processing policy to obtain the processed new target processing module.

[0171] In some embodiments, as shown in FIG. 17, in addition to the embodiment shown in FIG. 16 above, the image processing device 120 based on the system-on-chip further includes a sixth determination module 1209 and a fourth processing module 1210.

[0172] The sixth determination module 1209 is for determining the second processing result based on the processed new target processing module; The fourth processing module 1210 indicates that the second processing result has been successfully processed for the processed new target processing module, and in response to no abnormality information occurring during the processing of the third image data frame by the processed new target processing module, determines that the upgraded processing policy is valid and updates the abnormality processing correspondence relationship with the upgraded processing policy.

[0173] In some embodiments, as shown in FIG. 18, in addition to the embodiment shown in FIG. 12 above, the image processing device 120 based on the system-on-chip further includes a seventh determination module 1211 and a fifth processing module 1212.

[0174] The seventh determination module 1211 is for determining the number of times that abnormality information occurs in the abnormality module within a predetermined period of time; a fifth processing module 1212 for processing the target processing module based on a predetermined processing policy in response to the number of times being equal to or greater than the number of times threshold, and acquiring the processed target processing module; Here, the predetermined processing policy is an effective policy for resolving the abnormality information.

[0175] Regarding the image processing device using a system-on-chip in the above embodiment, the specific form of the execution operation of each module and the corresponding beneficial effects are described in detail in the corresponding embodiment of the image processing method using a system-on-chip described above. You can refer to the corresponding execution operation method and beneficial technical effects of the above exemplary method, and the description will be omitted here.

[0176] [Example Electronic Devices] FIG. 19 is a schematic diagram of an electronic device according to an exemplary embodiment of the present disclosure. As shown in FIG. 19, the electronic device 190 includes one or more processors 1901 and a memory 1902.

[0177] The processor 1901 may be a central processing unit (CPU) or other form of processing unit having data processing and / or instruction execution capabilities, and may control other components of the electronic device 190 to perform desired functions.

[0178] The memory 1902 may include one or more computer program products, which may include various types of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), a hard disk, flash memory, etc. The computer-readable storage media may store one or more computer program instructions, and the processor 1901 may execute the program instructions to implement the image processing method and / or other desired functions of the system-on-chip of each embodiment of the present disclosure.

[0179] In one example, electronic device 190 may further include input devices 1903 and output devices 1904 connected to each other via a bus system and / or other form of connection (not shown).

[0180] Of course, for simplicity, Figure 19 shows only some of the components in electronic device 190 that are relevant to the present disclosure, and omits components such as buses and input / output interfaces. Besides, electronic device 190 may further include any other appropriate components depending on the specific application.

[0181] Exemplary Computer Program Products and Computer-Readable Storage Media In addition to the methods and apparatus described above, embodiments of the present disclosure may be a computer program product including computer program instructions that, when executed by a processor, cause the processor to perform the steps of the system-on-chip image processing methods of various embodiments of the present disclosure described in the "Exemplary Methods" section above of this specification.

[0182] The computer program product may be written in any combination of one or more programming languages ​​to create program code for carrying out operations of embodiments of the present disclosure, including object-oriented programming languages ​​such as Java, C++, etc., and may further include general procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user equipment, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or a server.

[0183] Additionally, an embodiment of the present disclosure may also be a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, cause the processor to perform steps in the system-on-chip image processing methods of various embodiments of the present disclosure described in the "Exemplary Methods" section above of this specification.

[0184] The computer-readable storage medium may be any combination of one or more types of readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may include, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0185] Although the basic principles of the present disclosure have been described above with reference to specific embodiments, the benefits, advantages, effects, etc. mentioned in the present disclosure are merely illustrative and not limiting, and various embodiments of the present disclosure do not necessarily have these benefits, advantages, effects, etc. Furthermore, the specific details of the above disclosure are merely for the purpose of illustrative and easy-to-understand functions and are not limiting, and the details do not necessarily limit the present disclosure to be realized by the specific details.

[0186] Block diagrams of devices, apparatus, instruments, and systems related to this disclosure are merely illustrative examples and do not require or suggest that they must be connected, configured, or arranged in the manner shown in the block diagrams. Those skilled in the art will appreciate that these devices, apparatus, instruments, and systems can be connected, configured, and arranged in any manner. For example, terms such as "include," "comprise," and "have" are open terms and mean "including, but not limited to," and can be used interchangeably. As used herein, the terms "or" and "and" mean the term "and / or" and can be used interchangeably, unless otherwise specified. As used herein, the term "for example," means the term "including, but not limited to," and can be used interchangeably.

[0187] In the devices, apparatuses, and methods of the present disclosure, each component or each step can be disassembled and / or recombined, and such disassembly and / or recombination should be considered as an equivalent means of the present disclosure.

[0188] The previous description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Thus, the present disclosure is not intended to be limited to the aspects shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0189] The foregoing description has been provided for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of the present disclosure to the precise form disclosed herein. While several exemplary aspects and embodiments have been described above, those skilled in the art may recognize certain variations, modifications, variations, additions, and subcombinations thereof.

Claims

1. An image processing method using a system on a chip, comprising: determining abnormality information occurring during the process of processing a first image data frame by a plurality of image processing modules in the system-on-chip and path setting information corresponding to the plurality of image processing modules; Determining an abnormal module where the abnormal information occurs and an initial processing policy for handling the abnormal information; determining a target processing module from the plurality of image processing modules based on the abnormal module, the initial processing policy, and the path setting information; processing the target processing target module based on the initial processing policy; and processing a second image data frame with the processed target processing object module.

1. A system-on-chip image processing method comprising:

2. The step of determining an abnormal module where the abnormal information occurs and an initial processing policy for handling the abnormal information includes: Analyzing the anomaly information to determine the anomaly module, and a module identifier of the anomaly module and / or an anomaly type of the anomaly module; searching for an abnormality processing correspondence relationship based on the abnormal module, the module identifier, and / or the abnormality type, and determining a processing policy corresponding to the abnormal module, the module identifier, and / or the abnormality type as the initial processing policy; 2. The image processing method according to claim 1, wherein the image processing method is a system-on-chip method.

3. The step of determining a target processing module from the plurality of image processing modules based on the abnormal module, the initial processing policy, and the path setting information includes: determining at least one pass module from the plurality of image processing modules based on first mode setting information in response to the initial processing policy being one of frame drop, module reset, and partial reset and partial frame drop; determining the target processing module from at least one of the path modules based on the abnormal module and second mode setting information; At least one of the pass modules includes the abnormality module; the first mode setting information is for setting a connection method for each of the image processing modules, the second setting information is for setting an operation mode of each of the image processing modules, the path setting information includes the first mode setting information and the second mode setting information; 2. The image processing method according to claim 1, wherein the image processing method is a system-on-chip method.

4. The step of determining the target processing module from at least one of the path modules based on the abnormal module and second mode setting information includes: determining a containment relationship between a plurality of path modules downstream of the abnormal module and a self-processing module for intercepting and processing data according to the second mode setting information and at least one of the path modules; determining the target processing module from at least one of the path modules based on the inclusion relationship; 4. The image processing method using a system-on-chip according to claim 3.

5. The step of determining the target processing module from at least one of the path modules based on the inclusion relationship includes: determining the self-processing module as the target processing module in response to the initial processing policy being the frame drop and a plurality of path modules downstream of the abnormal module including the self-processing module; determining at least one of the path modules as the target processing module in response to the initial processing policy being the frame drop and a plurality of path modules downstream of the abnormal module not including the self-processing module; 5. The image processing method according to claim 4, wherein the image processing method is performed by a system-on-chip.

6. The step of determining the target processing module from at least one of the path modules based on the inclusion relationship includes: determining the self-processing module and the upstream path module of the self-processing module as the target processing target module in response to the initial processing policy being a reset of the module and the plurality of downstream path modules of the abnormal module including the self-processing module; determining at least one of the path modules as the target processing module in response to the initial processing policy being a reset of the module and a plurality of path modules downstream of the abnormal module not including the self-processing module; 5. The image processing method according to claim 4, wherein the image processing method is performed by a system-on-chip.

7. The step of determining the target processing module from at least one of the path modules based on the inclusion relationship includes: In response to the initial processing policy being the partial reset and partial frame drop and a plurality of path modules downstream of the abnormal module including the self-processing module, when resetting the path module, determining the self-processing module and a path module upstream of the self-processing module as the target processing target module, and when performing frame drop processing on the path module, determining the path module downstream of the self-processing module as the target processing target module.

7. The image processing method according to claim 6, wherein the image processing method is performed by a system-on-chip.

8. The step of determining the target processing module from at least one of the path modules based on the abnormal module and second mode setting information includes: determining the plurality of image processing modules as the target processing target modules in response to the initial processing policy being a system reset; 2. The image processing method according to claim 1, wherein the image processing method is a system-on-chip method.

9. The image processing method using the system-on-chip includes: after processing the target processed module based on the initial processing policy to obtain the processed target processed module; determining a first processing result based on the processed target processing module; determining whether the abnormality information occurs during the process of the processed target processing module processing the second image data frame in response to the first processing result indicating that the processing for the target processing module is successful; In response to the abnormal information occurring during the processing of the second image data frame by the processed target processed module, determining that the initial processing policy is invalid, upgrading the initial processing policy, determining a new target processed module corresponding to the upgraded processing policy, and processing the new target processed module according to the upgraded processing policy to obtain the processed new target processed module.

9. The image processing method using the system-on-chip according to claim 1.

10. The image processing method using the system-on-chip includes: After the step of processing the new target processing module according to the upgraded processing policy, determining a second processing result based on the processed new target processing module; determining that the upgraded processing policy is valid and updating the abnormality processing correspondence relationship according to the upgraded processing policy in response to the second processing result indicating that the processing for the processed new target processing module is successful and the abnormality information does not occur during the processing of the third image data frame by the processed new target processing module; 10. The image processing method according to claim 9, wherein the image processing method is a system-on-chip method.

11. The image processing method using the system-on-chip includes: determining the number of times the abnormality information occurs in the abnormal module within a predetermined period of time; In response to the number of times being equal to or greater than a number threshold, processing the target processing target module based on a predetermined processing policy to obtain the processed target processing target module; the predetermined processing policy is an effective policy for resolving the anomaly information; 9. The image processing method using the system-on-chip according to claim 1.

12. An image processing device using a system on a chip, a first determination module for determining abnormality information occurring during the process of processing a current image data frame by a plurality of image processing modules in the system-on-chip and path setting information corresponding to the plurality of image processing modules; a second determination module for determining an anomaly module where the anomaly information occurs and an initial processing policy for handling the anomaly information; a third determination module for determining a target processing module from the plurality of image processing modules based on the abnormal module, the initial processing policy, and the path setting information; a first processing module for processing the target processed module based on the initial processing policy to obtain the processed target processed module, and for processing a next image data frame using the processed target processed module; 1. A system-on-chip image processing device comprising:

13. A computer-readable storage medium, comprising: The storage medium stores a computer program for executing the image processing method using a system-on-chip according to any one of claims 1 to 8. A computer-readable storage medium comprising:

14. An electronic device, a processor; a memory for storing instructions executable by the processor; The processor reads the executable instructions from the memory and executes them to perform the image processing method using a system-on-chip according to any one of claims 1 to 8. An electronic device characterized by:

Images