Image processing method, device and chip by artificial intelligence accelerator
The image processing method using an AI accelerator in autonomous vehicles addresses hardware failure-induced errors by switching between storage spaces and accelerators, enhancing safety by reducing error probabilities and extending fault tolerance.
Patent Information
- Application Number
- JP2024075233
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-08
- Filing Date
- 2024-05-07
- Publication Date
- 2024-11-20
- Estimated Expiration
- 2044-05-07
AI Technical Summary
Existing artificial intelligence accelerators in autonomous driving systems are prone to hardware failures, leading to high probabilities of image processing errors, which compromise vehicle safety.
An image processing method using an artificial intelligence accelerator that determines algorithm model information and executes instructions from a first preset storage space, allowing for the use of different storage spaces or accelerators in a round-robin manner to reduce errors due to hardware failures.
This approach effectively reduces the probability of image processing errors and enhances vehicle safety by avoiding continuous errors through redundant storage and accelerator usage, extending fault tolerance time.
Smart Images

Figure 2024161907000001_ABST
Abstract
Description
[Technical field]
[0001] The present disclosure relates to the field of artificial intelligence chips, and in particular to an image processing method, apparatus and chip by an artificial intelligence accelerator. [Background technology]
[0002] In scenarios such as autonomous driving and driving assistance, it is usually necessary to use an artificial intelligence accelerator to accelerate an algorithm model so as to complete image processing such as detection and identification of images collected by a sensor to obtain a processing result. In the application process of the artificial intelligence accelerator of the related technology, the failure of the accelerator hardware or memory hardware is likely to cause an increased probability of image processing errors, which further reduces the safety of vehicle driving. Summary of the Invention [Problem to be solved by the invention]
[0003] In order to solve technical problems such as the high probability of image processing errors caused by the above-mentioned hardware failure, the embodiments of the present disclosure provide an image processing method, apparatus, and chip using an artificial intelligence accelerator, which effectively reduces the probability of image processing errors caused by hardware failure, and improves safety during vehicle operation. [Means for solving the problem]
[0004] An image processing method using an artificial intelligence accelerator according to a first aspect of the present disclosure includes the steps of: determining algorithm model information corresponding to an image to be processed; determining an artificial intelligence accelerator that needs to execute an algorithm model instruction corresponding to the algorithm model information based on the algorithm model information; and reading, by the artificial intelligence accelerator, the algorithm model instruction from a first pre-set memory space in which the algorithm model instruction is stored, and executing the algorithm model instruction to obtain a processing result for the image to be processed.
[0005] An image processing device using an artificial intelligence accelerator according to a second aspect of the present disclosure includes a first control module for determining algorithm model information corresponding to an image to be processed, a second control module for determining an artificial intelligence accelerator that needs to execute an algorithm model instruction corresponding to the algorithm model information based on the algorithm model information, and an artificial intelligence accelerator for reading the algorithm model instruction from a first pre-set memory space in which the algorithm model instruction is stored, and executing the algorithm model instruction to obtain a processing result for the image to be processed.
[0006] An artificial intelligence chip according to a third aspect of the present disclosure comprises a first pre-set memory space and an image processing device by an artificial intelligence accelerator, the image processing device by the artificial intelligence accelerator being for executing the image processing method by the artificial intelligence accelerator described in the first aspect above. Effect of the Invention
[0007] According to the method, device and chip for processing an image by an artificial intelligence accelerator according to the above embodiment of the present disclosure, when an image needs to be processed, the algorithm model information corresponding to the image to be processed is determined, and the artificial intelligence accelerator that needs to execute the algorithm model command corresponding to the algorithm model information is determined according to the algorithm model information, and the determined artificial intelligence accelerator reads and executes the algorithm model command from the first pre-defined memory space in which the algorithm model command is stored, so as to obtain the processing result of the image to be processed. In the image processing process, the algorithm model information corresponding to the image to be processed of different frames can be determined, so that in the process of calling the artificial intelligence accelerator of different frames, the algorithm model command of different memory spaces can be adopted or different artificial intelligence accelerators can be invoked to realize the processing of the image. When the memory space or accelerator hardware of any frame image is invalid, the image processing of the subsequent frame of the frame image can adopt the algorithm model command of another memory space or invoke another artificial intelligence accelerator, so as to avoid the occurrence of continuous errors, which can effectively reduce the probability of image processing errors caused by the failure of memory or accelerator hardware, and greatly improve the safety during the running of the vehicle. [Brief description of the drawings]
[0008] [Figure 1] 1 is an exemplary application scenario of an image processing method using an artificial intelligence accelerator according to the present disclosure. [Diagram 2] 1 is a flowchart of an image processing method by an artificial intelligence accelerator according to an exemplary embodiment of the present disclosure. [Diagram 3] 11 is a flowchart of an image processing method by an artificial intelligence accelerator according to another exemplary embodiment of the present disclosure. [Figure 4] 4 is a flowchart of an image processing method by an artificial intelligence accelerator according to a further exemplary embodiment of the present disclosure. [Diagram 5] 11 is a flowchart of an image processing method by an artificial intelligence accelerator according to a still further exemplary embodiment of the present disclosure. [Figure 6] FIG. 2 is a schematic diagram of accelerator scheduling for multi-frame image processing according to an exemplary embodiment of the present disclosure. [Figure 7] 4 is a flowchart of an image processing method by an artificial intelligence accelerator according to a further exemplary embodiment of the present disclosure. [Figure 8] 11 is a flowchart of an image processing method by an artificial intelligence accelerator according to a still further exemplary embodiment of the present disclosure. [Figure 9] 4 is a flowchart of an image processing method by an artificial intelligence accelerator according to a further exemplary embodiment of the present disclosure. [Figure 10] FIG. 1 is a structural schematic diagram of an image processing apparatus based on an artificial intelligence accelerator according to an exemplary embodiment of the present disclosure; [Figure 11] FIG. 2 is a structural schematic diagram of an image processing apparatus based on an artificial intelligence accelerator according to another exemplary embodiment of the present disclosure; [Figure 12] 1 is a structural schematic diagram of an artificial intelligence chip according to an exemplary embodiment of the present disclosure; FIG. [Figure 13] FIG. 1 is a structural diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0009] Hereinafter, exemplary embodiments according to the present disclosure will be described in detail with reference to the drawings. Obviously, the described embodiments are merely some of the embodiments of the present disclosure, but are not all of the embodiments, and the present disclosure is not limited to the described exemplary embodiments.
[0010] The relative arrangement of components or steps described in these examples, numerical expressions and values do not limit the scope of the present disclosure unless specifically stated otherwise.
[0011] (Summary of the Disclosure) In the process of realizing the present disclosure, the inventors have found that in scenes such as autonomous driving and driving assistance, it is usually necessary to use an artificial intelligence accelerator to accelerate an algorithm model so as to complete image processing such as detection and identification of images collected by a sensor to obtain a processing result. In the application process of the artificial intelligence accelerator of the related technology, the failure of the accelerator hardware or memory hardware is likely to cause a high probability of image processing errors, which further reduces the safety of vehicle driving.
[0012] (Illustrative Overview) 1 is an exemplary application scene of the image processing method by the artificial intelligence accelerator according to the present disclosure. When an image needs to be processed, the image processing method by the artificial intelligence accelerator according to the present disclosure (this method is realized by an image processing device by an artificial intelligence accelerator) is used for the image to be processed to determine algorithm model information corresponding to the image to be processed, including at least one of information such as identification information of the algorithm model instruction and the memory space address of the algorithm model instruction, and the artificial intelligence accelerator that needs to execute the algorithm model instruction corresponding to the algorithm model information based on the algorithm model information is determined, and the determined artificial intelligence accelerator reads the algorithm model instruction from the first preset memory space in which the algorithm model instruction is stored and executes the algorithm model instruction to obtain the processing result for the image to be processed. In the image processing process, the algorithm model information corresponding to the image to be processed of different frames can be determined, so that in the process of calling the artificial intelligence accelerator of different frames, the algorithm model instruction of different memory spaces can be adopted or different artificial intelligence accelerators can be called to realize the processing of the image. For example, the same algorithm model instructions are stored in multiple memory spaces of the memory, and algorithm model information corresponding to the algorithm model instructions of each memory space is set. During image processing, the algorithm model instructions of each memory space are adopted in a round-robin manner at a certain frame interval for image processing. Also, for example, multiple artificial intelligence accelerators are called in a round-robin manner at a certain frame interval for image processing. The specific frame interval can be set according to actual needs, for example, the frame interval may be 1 frame, 2 frames, 3 frames, etc.Based on this, when the memory space or accelerator hardware of any frame image is invalid, the image processing of the subsequent frames of the frame image can adopt the algorithm model instructions of other memory spaces or call other artificial intelligence accelerators, thereby avoiding the continuous occurrence of image processing errors, thereby effectively reducing the probability of image processing errors caused by the expiration of memory or accelerator hardware, and greatly improving the safety of the vehicle during driving.
[0013] The image processing method using an artificial intelligence accelerator according to the embodiment of the present disclosure can be applied to fields or scenes in which images need to be processed by any artificial intelligence accelerator, such as autonomous driving scenes, driving assistance scenes, and application scenes of terminal devices such as mobile phones and tablet computers, and the specific fields and scenes are not limited.
[0014] Exemplary Methods 2 is a flowchart of an image processing method by an artificial intelligence accelerator according to an exemplary embodiment of the present disclosure. This embodiment can be applied to an electronic device, specifically, for example, an in-vehicle computing platform, and includes the following steps 201 to 203, as shown in FIG.
[0015] In step 201, algorithm model information corresponding to the image to be processed is determined.
[0016] Here, the algorithm model information may include at least one of the following information: identification information of the algorithm model command, the memory space address of the algorithm model command, the name of the algorithm model, etc. The algorithm model information corresponding to the image to be processed of the algorithm model may be determined according to a mapping rule set in advance. The mapping rule may be set according to actual needs.
[0017] In some selectable embodiments, the algorithm model information can include related information corresponding to at least one type of algorithm model. For example, for an autonomous driving scene, the algorithm model information can include a forward vision detection model, a peripheral vision detection model, a DMS (Driver Monitor System) detection model, a detection fusion model, a prediction model, a planning model, a diagnostic model, a laser radar detection model, etc. The processing of the target image can include direct processing based on the detection model and subsequent application of the detection result based on the planning model, etc., so that complete vehicle planning and control can be realized based on the target image, and can be specifically set according to actual needs.
[0018] In some alternative embodiments, the image to be processed can include images of one or more viewing angles of the current frame, for example, a forward viewing image, a peripheral viewing image, and the images of different viewing angles can correspond to different algorithm models. The algorithm model information includes related information of a group of algorithm model instructions required for the current image processing.
[0019] In some alternative embodiments, the mapping rule can be set based on the frame information of the image to be processed, for example, for the same group of algorithm models, the algorithm model instructions can be copied to multiple parts and stored in different storage sub-spaces in the first preset storage space (for example, memory), and the algorithm model information corresponding to the algorithm model instructions of each part can be set and used to uniquely identify the algorithm model instructions of different parts, to obtain multiple algorithm model information, and a mapping rule between the frame information and the algorithm model information can be established, for example, corresponding to each algorithm model information in a round-robin manner at a preset frame interval. Take three algorithm model information as an example, when the preset frame interval is 1, the n+1th frame corresponds to the algorithm model information 1, the n+2th frame corresponds to the algorithm model information 2, and the n+3rd frame corresponds to the algorithm model information 3, where n=0, 1, . . . The preset frame interval can also be set to 2, 3, 4, etc., and is not specifically limited.
[0020] In some alternative embodiments, for algorithmic models of the same group, only one copy may be stored in the first pre-defined storage space.
[0021] In step 202, based on the algorithm model information, an artificial intelligence accelerator that needs to execute the algorithm model instruction corresponding to the algorithm model information is determined.
[0022] Here, the number of artificial intelligence accelerators (which can be abbreviated as "accelerators") can be one or more. The artificial intelligence accelerator that needs to execute the algorithm model instruction corresponding to the algorithm model information can be determined according to a preset rule. The specific rule can be set according to actual needs.
[0023] In some selectable embodiments, the number of preset algorithm model information is multiple, and the number of artificial intelligence accelerators for executing the algorithm model instructions corresponding to the preset algorithm model information can be one or multiple. If the number of selectable artificial intelligence accelerators is one, the current algorithm model information can be determined according to the image to be processed, and the artificial intelligence accelerator can be determined directly to execute the algorithm model instructions corresponding to the algorithm model information based on the algorithm model information. If the number of selectable artificial intelligence accelerators is multiple, the accelerator for executing the current algorithm model instructions can be determined from the multiple artificial intelligence accelerators according to a preset rule. For example, multiple accelerators are called in a round-robin manner at frame intervals, or multiple accelerators are called simultaneously for each frame to execute different parts of the algorithm model instructions, and the specific can be set according to actual needs, as long as it can effectively reduce the continuous errors in image processing.
[0024] In some selectable embodiments, the number of pre-set algorithm model information is one, the number of selectable accelerators is one, and different calculation units in the accelerator are called in a round-robin manner at frame intervals to execute the algorithm model instructions, thereby reducing continuous errors caused by hardware failures of the calculation units, thereby reducing continuous errors in image processing, reducing the probability of image processing errors caused by accelerator hardware failures, and further improving the safety of vehicle driving.
[0025] In some optional embodiments, the number of preset algorithm model information is one, and the number of preset selectable accelerators is multiple, and the algorithm model information corresponding to each frame is the same. According to a preset rule, an accelerator for executing the current algorithm model instruction can be determined from the multiple artificial intelligence accelerators, for example, calling multiple accelerators in a round-robin manner at frame intervals or calling multiple accelerators simultaneously for each frame to execute different parts of the algorithm model instruction, and the specific setting can be made according to actual needs.
[0026] In some alternative embodiments, when each frame calls multiple accelerators simultaneously to execute different parts of the algorithm model instructions, different parts of the algorithm model instructions can also be executed in a round-robin manner in different frames for each accelerator, for example, by dividing the algorithm model instructions into multiple parts according to the number of algorithm models, for example including algorithm part 1, algorithm part 2, and algorithm part 3, and having three accelerators, in the n+1th frame, accelerator 1 executes algorithm part 1, accelerator 2 executes algorithm part 2, and accelerator 3 executes algorithm part 3; in the n+2th frame, accelerator 1 executes algorithm part 2, accelerator 2 executes algorithm part 3, and accelerator 3 executes algorithm part 1; and in the n+3th frame, accelerator 1 executes algorithm part 3, accelerator 2 executes algorithm part 1, and accelerator 3 executes algorithm part 2.
[0027] In step 203, the artificial intelligence accelerator reads the algorithm model instruction from the first pre-defined memory space in which the algorithm model instruction is stored, and executes the algorithm model instruction to obtain a processing result for the target image.
[0028] Here, after determining the artificial intelligence accelerator, the memory space address of the algorithm model instruction is sent to the accelerator, so that the accelerator reads and executes the algorithm model instruction from the first pre-defined memory space in which the algorithm model instruction is stored, and finally obtains the processing result.
[0029] In some optional embodiments, when multiple accelerators are executing, each accelerator reads and executes a portion of the algorithm model instructions that it should execute to obtain its own processing result, and obtains a processing result for the image to be processed based on the processing results of the multiple accelerators.
[0030] In some optional embodiments, the artificial intelligence accelerator includes a computing unit used for various operations in the algorithm model, and the algorithm model instructions control the calculation of the computing unit to realize the corresponding operations of the algorithm model to obtain a processing result of the image to be processed.
[0031] In some optional embodiments, model parameters such as weights and offsets required for the accelerator's inference can also be stored in the first preset memory space or other memory space, and the accelerator can read the corresponding model parameters and execute the algorithm model instructions to complete the inference calculation of the algorithm model.
[0032] In some alternative embodiments, the first pre-defined storage space can be a memory space.
[0033] In the method for processing an image by an artificial intelligence accelerator according to the present embodiment, when an image needs to be processed, the algorithm model information corresponding to the image to be processed is determined, and the artificial intelligence accelerator that needs to execute the algorithm model command corresponding to the algorithm model information is determined according to the algorithm model information, and the determined artificial intelligence accelerator can read and execute the algorithm model command from the first preset memory space in which the algorithm model command is stored to obtain the processing result of the image to be processed. In the image processing process, the corresponding algorithm model information can be determined for the images to be processed of different frames, so that in the process of calling the artificial intelligence accelerators of different frames, the same algorithm model command in different memory spaces is adopted or different artificial intelligence accelerators are invoked to process the image, and when the memory space or accelerator hardware of any frame image is faulty, the image processing of the subsequent frames of the frame image can adopt the algorithm model command in another memory space or invoke another artificial intelligence accelerator, so as to avoid the occurrence of continuous errors, which can effectively reduce the probability of image processing errors caused by memory or accelerator hardware faults, and greatly improve the safety during the running of the vehicle.
[0034] FIG. 3 is a flowchart of an image processing method by an artificial intelligence accelerator according to another exemplary embodiment of the present disclosure.
[0035] In some alternative embodiments, step 201 of determining algorithm model information corresponding to the image to be processed includes the following steps 2011 to 2012.
[0036] In step 2011, the current frame information corresponding to the image to be processed is determined.
[0037] Here, the current frame information may include a frame number to which the image to be processed belongs. For example, the current frame information may include that the image to be processed is the Nth frame.
[0038] In some optional embodiments, for example in an autonomous driving scene, images usually need to be processed continuously, and frame information of the processed image can be recorded and maintained in real time from the start of image processing. When an image to be processed is obtained, current frame information corresponding to the image to be processed can be determined based on the frame information maintained in real time.
[0039] In step 2012, algorithm model information corresponding to the current frame information is determined from the plurality of algorithm model information according to a mapping rule between frame information and algorithm model information.
[0040] Here, the plurality of algorithm model information respectively correspond to the same algorithm model command in different storage sub-spaces in the first preset storage space. The mapping rule between the frame information and the algorithm model information can be set according to actual needs, for example, establish a correspondence relationship between each frame and the algorithm model information at a preset frame interval. The preset frame interval can be set according to actual needs, and is not limited in the present disclosure.
[0041] In some selectable examples, if the preset frame interval is 1 frame and the number of algorithm model information is m, the first frame corresponds to algorithm model information 1, the second frame corresponds to algorithm model information 2, the third frame corresponds to algorithm model information 3, ..., the mth frame corresponds to algorithm model information m, the m+1th frame corresponds to algorithm model information 1, the m+2th frame corresponds to algorithm model information 2, ..., the 2mth frame corresponds to algorithm model information m, and so on.
[0042] In some selectable examples, when the preset frame interval is 2 frames and the number of algorithm model information is m, the first frame corresponds to algorithm model information 1, the second frame corresponds to algorithm model information 1, the third frame corresponds to algorithm model information 2, the fourth frame corresponds to algorithm model information 2, ..., the 2m-1th frame corresponds to algorithm model information m, the 2mth frame corresponds to algorithm model information m, the 2m+1th frame corresponds to algorithm model information 1, ...
[0043] In this embodiment, through the mapping rules between frame information and algorithm model information, the images to be processed of different frames can adopt algorithm model instructions of different memory sub-spaces. If an error occurs in the algorithm model instructions of any frame image due to a failure of the memory space hardware, the image processing of the subsequent frames of the frame image can adopt the algorithm model instructions of other memory sub-spaces. Therefore, the continuity error caused by the failure of the memory space hardware can be effectively avoided, and the continuous error can be changed into an accidental or intermittent error, which greatly extends the fault-tolerant time and is useful for further improving the safety of the vehicle during driving in combination with error detection.
[0044] In some alternative embodiments, the following steps 310 to 320 are further included before step 201 of determining algorithm model information corresponding to the image to be processed.
[0045] In step 310, read the algorithm model instructions required for image processing from the second preset storage space based on the preset information.
[0046] Here, the pre-defined information may be set according to actual function needs, for example, in an automatic driving scene, the function of planning and control is executed based on the detection result, and the pre-defined information may include related information of algorithm models required for image processing of planning and control, for example, related information of algorithm models such as forward vision detection model, peripheral vision detection model, DMS (Driver Monitor System) detection model, detection fusion model, prediction model, planning model, diagnosis model, etc. The related information may include storage addresses in the second pre-defined storage space of algorithm model commands corresponding to each algorithm model and other related information, and may be specifically set according to actual needs. The pre-defined information may further include related operations (e.g., read operation, write operation, number of writes, etc.) that need to be performed, so that the algorithm model commands of the algorithm models required for image processing can be read from the corresponding addresses in the second pre-defined storage space based on the pre-defined information.
[0047] In some alternative embodiments, the second pre-defined storage space can be a storage space in an external memory, such as a hard disk, a floppy disk, an optical disk, etc.
[0048] In step 320, the algorithmic model instructions are written to a number of different storage sub-spaces of a first pre-defined storage space.
[0049] Here, after reading an algorithm model command required for image processing, the algorithm model command can be written into multiple different memory sub-spaces of the first preset memory space based on writing information such as the write operation and the number of writes in the preset information, so that multiple copies of the same algorithm model command can be stored in the first preset memory space; and, based on the mapping rule between frame information and algorithm model information, the algorithm model information corresponding to the current frame information is determined from the multiple algorithm model information; an accelerator for the algorithm model command corresponding to the execution of the algorithm model information is determined; and then the algorithm model command can be read and executed from the memory sub-space corresponding to the algorithm model information in the first preset memory space by the corresponding accelerator.
[0050] In this embodiment, the algorithm model instructions of one group of algorithm models required for image processing are read from the second preset memory space and written into multiple different memory sub-spaces of the first preset memory space, thereby storing multiple copies of the same algorithm model instructions in the first preset memory space, and providing effective data support for the round-robin use of subsequent algorithm model instructions, so that when the hardware of any memory sub-space fails, the algorithm model instructions of other memory sub-spaces are subsequently adopted, thereby effectively avoiding continuous errors caused by the hardware failure of some memory sub-spaces, and thereby changing continuous errors into intermittent errors, which is helpful in effectively extending the fault-tolerant time and further improving the coverage rate of error detection.
[0051] FIG. 4 is a flowchart of an image processing method by an artificial intelligence accelerator according to a further exemplary embodiment of the present disclosure.
[0052] In some optional embodiments, step 202 of determining an artificial intelligence accelerator that needs to execute an algorithm model instruction corresponding to the algorithm model information based on the algorithm model information includes the following steps 2021 to 2022.
[0053] In step 2021, an accelerator alternating scheduling rule corresponding to the algorithm model information is determined.
[0054] Here, the accelerator alternating scheduling rule refers to a rule for alternating scheduling of multiple accelerators, i.e., the number of selectable accelerators is multiple, and the multiple accelerators are scheduled based on the accelerator alternating scheduling rule and used to execute the algorithm model instructions corresponding to the algorithm model information. The accelerator alternating scheduling rule corresponding to the algorithm model information can be determined based on a pre-defined correspondence. For example, different accelerator alternating scheduling rules can be set for different algorithm model groups. Thus, the accelerator alternating scheduling rule corresponding to the algorithm model group can be determined based on the algorithm model information.
[0055] In some alternative embodiments, the accelerator alternating scheduling rule may include alternating scheduling of multiple accelerators at a preset frame interval, and the preset frame interval may be set according to actual needs. For example, if the preset frame interval is 1, the first frame corresponds to scheduling accelerator 1, the second frame corresponds to scheduling accelerator 2, the nth frame corresponds to scheduling accelerator n, the n+1th frame corresponds to scheduling accelerator 1, the n+2th frame corresponds to scheduling accelerator 2, and so on, where n is the number of accelerators.
[0056] In some alternative embodiments, the accelerator alternating scheduling rule may include a scheduling order for multiple accelerators and may further include the number of accelerators to be scheduled each time, for example, the scheduling order is accelerator 1-accelerator 2-accelerator 3-accelerator 1-accelerator 2-accelerator 3-...
[0057] In step 2022, according to the accelerator alternating scheduling rule, determine an AI accelerator that needs to execute the algorithm model instruction corresponding to the current algorithm model information from among the multiple AI accelerators.
[0058] Here, the number of artificial intelligence accelerators that currently need to execute the algorithm model instructions corresponding to the algorithm model information may be one or more, and may be specifically set according to actual needs.
[0059] In some optional embodiments, one accelerator can be scheduled for each frame at a preset frame interval to execute the algorithm model instructions corresponding to the algorithm model information, for example, accelerator 1 can be scheduled for the t+1th frame, accelerator 2 can be scheduled for the t+2th frame, ..., accelerator n can be scheduled for the t+nth frame, where t=0, 1, 2, ..., and n is the number of accelerators.
[0060] In some selectable embodiments, at least two accelerators may be scheduled in each frame at a preset frame interval to jointly execute the algorithm model instructions. For example, the algorithm model instructions may be divided into algorithm model instructions corresponding to a plurality of algorithm model groups based on the algorithm models included therein, and each accelerator executes the algorithm model instructions corresponding to one algorithm model group. The at least two accelerators may include some or all of a plurality of selectable accelerators. For example, the number of selectable accelerators may be six, and two accelerators may be scheduled in each frame to jointly execute the algorithm model instructions of the two divided algorithm model groups, and the accelerators may be alternately scheduled at a preset frame interval, for example, accelerator 1 and accelerator 2 may be scheduled in the first frame, accelerator 3 and accelerator 4 may be scheduled in the second frame, accelerator 5 and accelerator 6 may be scheduled in the third frame, accelerator 1 and accelerator 2 may be scheduled in the fourth frame, and so on. Or, accelerator 1 and accelerator 2 are scheduled in the first frame, accelerator 2 and accelerator 3 are scheduled in the second frame, accelerator 3 and accelerator 4 are scheduled in the third frame, and so on. Other alternating scheduling rules can be adopted as long as they ensure that the number of times each accelerator consecutively executes the same algorithm model instruction does not exceed a preset number. The specific alternating scheduling rule is not limited.
[0061] This embodiment can realize alternating scheduling of multiple accelerators based on the accelerator alternating scheduling rule, so that each accelerator does not execute the same algorithm model instruction multiple times in succession, thereby effectively reducing continuous errors caused by accelerator hardware failures, changing continuous errors caused by accelerator hardware failures into intermittent errors, greatly extending the fault-tolerant time, and providing effective time support for accelerator hardware error detection, thereby helping to further improve the safety of the vehicle during driving.
[0062] In some alternative embodiments, the step 2022 of determining an artificial intelligence accelerator that needs to execute an algorithm model instruction corresponding to the current algorithm model information from among the multiple artificial intelligence accelerators according to an accelerator alternating scheduling rule may include: The method includes a step of determining an accelerator alternation order based on an accelerator alternation scheduling rule, and a step of determining an artificial intelligence accelerator that needs to execute an algorithm model instruction corresponding to the current algorithm model information based on the accelerator alternation order.
[0063] Here, the accelerator alternation sequence can be set according to actual needs. For example, the number of accelerators is 3, and the accelerator alternation sequence is accelerator 1-accelerator 2-accelerator 3, which indicates that accelerators 1 to 3 are cyclically alternating.
[0064] In some alternative embodiments, the number of artificial intelligence accelerators that need to execute the algorithm model instruction corresponding to the current algorithm model information determined according to the accelerator alternating order can be one or more.For multiple cases, the algorithm model instruction is processed by multiple accelerators in a cooperative manner, for details, please refer to the above content, and the description will be omitted here.
[0065] In this embodiment, by determining the alternating order of the accelerator, the artificial intelligence accelerator that needs to execute the algorithm model instruction corresponding to the current algorithm model information can be effectively determined based on the order, thereby providing an effective guarantee for the alternating scheduling of the accelerator.
[0066] FIG. 5 is a flowchart of an image processing method by an artificial intelligence accelerator according to yet a further exemplary embodiment of the present disclosure.
[0067] In some optional embodiments, step 2022 of determining an artificial intelligence accelerator that needs to execute an algorithm model instruction corresponding to current algorithm model information from a plurality of artificial intelligence accelerators based on an accelerator alternating scheduling rule includes the following steps 20221 to 20224.
[0068] In step 20221, identification information corresponding to each of the plurality of algorithm models is determined based on the algorithm model information.
[0069] Here, the identification information corresponding to the algorithm model may be set according to actual needs, and may be uniquely identified, and is not limited in the present disclosure. For example, the identification information of the algorithm model may be the name of the algorithm model or an ID set to the algorithm model. The algorithm model information may include the identification of the entire algorithm model command of one group of algorithm models required for image processing, and may also include the identification information of each algorithm model, the storage space address of the algorithm model command corresponding to each algorithm model, and other related information, and may be specifically set according to actual needs. Based on the algorithm model information, the identification information corresponding to each algorithm model included therein may be determined.
[0070] In step 20222, the accelerator alternation order is determined based on the accelerator alternation scheduling rule.
[0071] In step 20223, a plurality of algorithm model groups corresponding to the algorithm model information are determined based on the identification information corresponding to each algorithm model and the preset grouping rules.
[0072] Here, each algorithm model group includes at least one algorithm model identification information. The preset grouping rule can be set according to actual needs. For example, the preset grouping rule can be set randomly, or can be set according to the principle of load balancing based on the computing power required by each algorithm model, so as to ensure the load balancing of the accelerator and improve the processing efficiency. In addition, the plurality of algorithm model groups corresponding to the algorithm model information can be determined by any other possible embodiment, and is not specifically limited.
[0073] For example, the algorithm models included in the algorithm model information include a forward vision detection model, a peripheral vision detection model, a DMS detection model, a detection fusion model, a prediction model, a planning model, a diagnostic model and a laser radar detection model, and are divided into three algorithm model groups, where algorithm model group 1 includes the aforementioned detection model, peripheral vision detection model and laser radar detection model, algorithm model group 2 includes the detection fusion model and the DMS detection model, and algorithm model group 3 includes the prediction model, the planning model and the diagnostic model.
[0074] In step 20224, for any algorithm model group, based on the historical accelerator scheduling information corresponding to the algorithm model group, determine the artificial intelligence accelerator that currently needs to execute the algorithm model instruction corresponding to the algorithm model group according to the accelerator alternating order.
[0075] Here, the historical accelerator scheduling information includes the accelerator scheduling status of the previous frame, and can be used to determine the scheduling accelerator position of the current frame.
[0076] In some alternative embodiments, different initial accelerators of scheduling can be set for different algorithm model groups, for example, in the first frame, algorithm model group 1 corresponds to scheduling accelerator 1, algorithm model group 2 corresponds to scheduling accelerator 2, algorithm model group 3 corresponds to scheduling accelerator 3, and so on. In the process of processing subsequent frames, each algorithm model group can alternately schedule each accelerator according to the same accelerator alternating order, thereby ensuring that algorithm model instructions of multiple algorithm model groups are jointly completed by multiple accelerators in any one frame, and each algorithm model group can alternately schedule each accelerator at a preset frame interval. For example, based on the above-mentioned history accelerator scheduling information of the first frame, in the second frame, taking three accelerators as an example, according to the alternating order of accelerators 1-2-3, algorithm model group 1 should schedule accelerator 2, algorithm model group 2 should schedule accelerator 3, and algorithm model group 3 should schedule accelerator 1. In the third frame, algorithm model group 1 should schedule accelerator 3, algorithm model group 2 should schedule accelerator 1, algorithm model group 3 should schedule accelerator 2, and so on.
[0077] In this embodiment, by determining a plurality of algorithm model groups corresponding to the algorithm model information, the algorithm model command corresponding to the algorithm model information is divided into algorithm model commands corresponding to a plurality of algorithm model groups, and according to the accelerator alternating order, the plurality of accelerators are scheduled to respectively execute the algorithm model commands corresponding to each algorithm model group, so as to reduce the probability of image processing errors caused by the accelerator hardware, and improve the resource utilization rate of the accelerator, thereby effectively improving the image processing efficiency.
[0078] In some optional embodiments, the above-mentioned multiple algorithm model information and multiple algorithm model groups corresponding to each algorithm model information are combined, and multiple accelerators are scheduled to alternately execute algorithm model instructions corresponding to each algorithm model group, respectively, so as to realize that both the continuous error of memory hardware failure and the continuous error of accelerator hardware failure are changed to intermittent errors, and the fault-tolerant time is further extended, and effective time support is provided for memory hardware and accelerator hardware error detection, thereby reducing the probability of image processing errors, which is helpful in further improving the safety of the vehicle during driving by combining the error detection of memory hardware and accelerator hardware.
[0079] Illustratively, Figure 6 is a schematic diagram of accelerator scheduling for multi-frame image processing according to an exemplary embodiment of the present disclosure. In this example, the number of algorithm model information is 3, for example, algorithm model information 1, algorithm model information 2 and algorithm model information 3, that is, the same algorithm model instructions are respectively stored in three different storage sub-spaces (including storage sub-space 1 corresponding to algorithm model information 1, storage sub-space 2 corresponding to algorithm model information 2 and storage sub-space 3 corresponding to algorithm model information 3) of the first pre-defined storage space, the number of algorithm model groups corresponding to the algorithm model information is M, and the number of accelerators is M. In the Nth frame, the algorithm model instructions of the storage sub-space 1 corresponding to the algorithm model information 1 can be adopted to determine M algorithm model groups corresponding to the algorithm model information 1, including algorithm model group 1 to algorithm model group M, and schedule M accelerators according to the accelerator scheduling order, where algorithm model group 1 corresponds to scheduling accelerator 1, algorithm model group 2 corresponds to scheduling accelerator 2, and so on. In the N+1th frame, the algorithm model instruction of memory subspace 2 corresponding to algorithm model information 2 is adopted, and the history accelerator scheduling information in the Nth frame of algorithm model group 1 is the corresponding scheduling accelerator 1, so according to the accelerator scheduling order, in the N+1th frame, algorithm model group 1 corresponds to scheduling accelerator 2, similarly, algorithm model group 2 corresponds to scheduling accelerator 3, ..., algorithm model group M-1 corresponds to scheduling accelerator M, and algorithm model group M corresponds to scheduling accelerator 1.Similarly, in the N+2th frame, the algorithm model instructions in the memory subspace 3 corresponding to the algorithm model information 3 are adopted, and the algorithm model group 1 corresponds to the scheduling accelerator 3, the algorithm model group 2 corresponds to the scheduling accelerator 4, ..., the algorithm model group M corresponds to the scheduling accelerator 2. The subsequent frames are similar and the description is omitted.
[0080] The method for processing images using an artificial intelligence accelerator according to an embodiment of the present disclosure alternately adopts algorithm model instructions of different memory spaces at frame intervals, divides each group of algorithm model instructions into algorithm model instructions corresponding to a plurality of algorithm model groups, alternately schedules a plurality of accelerators according to an accelerator alternating order to jointly process the algorithm model instructions corresponding to a plurality of algorithm model groups, and checks the redundant configuration of the accelerator in the related technology. The present disclosure can greatly improve the effective utilization rate of accelerator resources, greatly improve processing performance, and effectively reduce costs without improving the usage of memory bandwidth.
[0081] In some optional embodiments, step 203 of reading the algorithm model instructions from a first pre-defined memory space that stores the algorithm model instructions by the artificial intelligence accelerator, executing the algorithm model instructions, and obtaining a processing result for the image to be processed includes the following steps 2031 to 2032.
[0082] In step 2031, for an artificial intelligence accelerator that needs to execute an algorithm model instruction corresponding to any algorithm model group, the artificial intelligence accelerator reads the algorithm model instruction corresponding to the algorithm model group from the first pre-set memory space, executes the algorithm model instruction, and obtains a processing result corresponding to the algorithm model group.
[0083] Wherein, different algorithm model groups can have different functions, and the accelerator reasoning can obtain the processing results corresponding to each algorithm model group. For any algorithm model group, the accelerator can read the algorithm model instruction corresponding to the algorithm model group from the corresponding memory sub-space in the first preset memory space according to the memory space address of the algorithm model instruction corresponding to the algorithm model group.
[0084] In some selectable embodiments, for any algorithm model group, the algorithm models included therein may be independent of each other or may have a dependency relationship, and are not specifically limited. For example, in an autonomous driving scene, the forward vision detection model and the peripheral vision detection model are independent of each other, and the detection fusion model needs to fuse various detections, so it can have a dependency relationship with the detection model. The processing results corresponding to the algorithm model group can include the processing results corresponding to each algorithm model, and can also include the processing results corresponding to the last algorithm model having a dependency relationship, and can be specifically set according to actual needs.
[0085] In some alternative embodiments, after each accelerator executes the corresponding algorithm model instruction, it can return the processing result to a preset memory space, for example, to a first preset memory space.
[0086] In step 2032, a processing result for the image to be processed is determined based on the processing results corresponding to each algorithmic model group.
[0087] Here, different algorithm model groups are used to complete different processing functions, and the processing results of each algorithm model group are integrated to obtain the processing result of the target image.
[0088] In some alternative embodiments, for a situation where multiple accelerators work together, after the inference of one of the accelerators (e.g., accelerator A) is completed, the processing result can be returned to a preset memory space, and when another accelerator (e.g., accelerator B) needs to perform subsequent processing based on the inference result of accelerator A, the other accelerator B can read the inference result of accelerator A from the preset memory space and perform further inference to obtain the inference result of accelerator B. In addition, communication between the accelerators can be established, thereby transmitting the inference result of accelerator A to accelerator B, which is not specifically limited.
[0089] This embodiment integrates the processing of a plurality of accelerators to effectively obtain a processing result corresponding to an image to be processed, thereby improving image processing efficiency.
[0090] In some alternative embodiments, the method further includes the following steps 410 to 420 before step 201 of determining algorithm model information corresponding to the image to be processed.
[0091] In step 410, read the algorithm model instructions required for image processing from the second preset storage space based on the preset information.
[0092] Here, the pre-set information can refer to the above-mentioned content, and the number of times of writing can be different from the above-mentioned content. In this embodiment, the number of times of writing is one.
[0093] In step 420, the algorithm model instructions are written to a first pre-defined memory space.
[0094] Here, the algorithm model command is written into the first preset storage space based on the write operation and the number of writes of the preset information.
[0095] This embodiment can copy algorithm model information from the second preset memory space to the first preset memory space in advance, so that the accelerator can quickly read and execute the algorithm model instructions, and improve the data reading speed.
[0096] FIG. 7 is a flowchart of an image processing method by an artificial intelligence accelerator according to a further exemplary embodiment of the present disclosure.
[0097] In some alternative embodiments, the first preset storage space stores a plurality of algorithm model instructions and model parameters corresponding to each algorithm model instruction. The method of the present disclosure further includes the following steps 430 to 440.
[0098] In step 430, check the preset contents in the plurality of algorithm model instructions in the first preset storage space and the model parameters corresponding to each algorithm model instruction respectively according to a preset period, and obtain a checking result.
[0099] Here, the preset period may be set according to actual needs, and is not specifically limited. The preset content may be set according to actual needs, for example, the preset content may be all the contents of the algorithm model instructions and model parameters, and may be the pre-set important contents among them, for example, the output layer of each algorithm model has a large impact on the algorithm model inference result, so the instructions and / or parameters corresponding to the output layer may be the important contents. The check result may include two types of results, that is, a successful check and a failed check, and the failed check result may include error content information (e.g., an address where an error occurs) that causes the failure. The check method may be to compare multiple parts of the algorithm model instructions and corresponding model parameters to determine whether the multiple parts are the same, and if there is a different content, it may be determined that an error occurs in the preset content.
[0100] In step 440, in response to the check result, a repair process is performed on the preset content based on the preset repair rule to obtain a repair result.
[0101] Here, the preset repair rule can be set according to actual needs. If the check result shows that the preset content has an error, the preset content is repaired according to the preset repair rule to obtain a repair result. For example, the correct content can be obtained from the correct part of the multiple preset contents, and the error content can be repaired. Also, the correct content repair error content can be read again from the second preset storage space.
[0102] In some optional embodiments, when repair is performed, the correct content after repair can be written to a different space than the erroneous content to avoid re-occurrence of the erroneous content due to storage hardware failure.
[0103] In this embodiment, by checking the algorithm model instructions and model parameters in the first pre-set memory space according to a certain period and repairing them in a timely manner, subsequent image processing errors caused by error content can be effectively avoided, so as to further improve the safety of vehicle driving.
[0104] In some alternative embodiments, step 430 includes: The method includes: checking, according to a first preset period, a plurality of algorithm model instructions in a first preset memory space and model parameters corresponding to each algorithm model instruction, respectively, to obtain a first checking result; and, in response to the first checking result, performing a repair process on the preset content based on a first repair rule, to obtain a first repair result.
[0105] Specifically, all contents of the multiple parts of algorithm model instructions and model parameters are checked according to a first preset period, and repair processing is performed based on the check result. For example, if an error exists in one or a few parts of the multiple parts, the correct content can be obtained from the correct part to repair the error content. If it is not possible to determine which part has an error, the algorithm model instructions and corresponding model parameters can be read again from the second preset storage space, and the multiple parts of algorithm model instructions and model parameters in the first preset storage space can be repaired.
[0106] In some alternative embodiments, the method of the present disclosure comprises: The method further includes a step of checking at least one of the preset key instructions and the preset key parameters corresponding to the multiple parts of algorithm model instructions in the first preset memory space according to a second preset period smaller than the first preset period to obtain a second checking result; and a step of repairing the error content based on the second checking result and a second repair rule to obtain a second repair result.
[0107] Here, the preset key command and the preset key parameter can be set according to the degree of influence on the algorithm model inference result in the algorithm model. For example, when an error occurs in the output layer or model parameter of the output layer of the algorithm model, it will cause serious error of the output result; when an error occurs in the previous model layer or model parameter in the algorithm model, the influence of the previous error can be continuously reduced in the subsequent inference process, so that the final inference result is accurate and no serious error occurs; based on this, the second preset period is set to be smaller than the first preset period, so that the error of the key content can be found more timely, so that the probability of image processing error caused by the failure of storage hardware can be further reduced, and the safety of vehicle driving can be further improved.
[0108] An embodiment of the present disclosure can quickly and frequently check essential content using a second preset period, and can check all content or non-essential content at a low speed using a first preset period, thereby effectively balancing resource occupancy and error checking problems. When occupying relatively low resources, errors in tolerant content can be quickly and timely detected, and the impact on inference results for other content can be small, and the resource occupancy can be reduced by checking at a lower frequency.
[0109] FIG. 8 is a flowchart of an image processing method by an artificial intelligence accelerator according to yet a further exemplary embodiment of the present disclosure.
[0110] In some optional embodiments, step 440 of performing a repair process on the preset content based on the preset repair rule in response to the check result to obtain a repair result includes the following steps 4410 to 4420.
[0111] In step 4410, in response to the check result, the algorithm model instructions and model parameters in which the error content exists are deleted from the first pre-set storage space, and the algorithm model instructions and model parameters corresponding to the deleted content are re-read from the second pre-set storage space.
[0112] Wherein, some or several algorithm model instructions and model parameters in which erroneous contents exist can be deleted from the first preset storage space, and reading the erroneous algorithm model instructions or model parameters later can be avoided. The algorithm model instructions and model parameters corresponding to the deleted contents are read again from the second preset storage space according to the preset information.
[0113] In step 4420, the re-read algorithmic model instructions and model parameters are written to a target subspace in the first pre-defined storage space.
[0114] Here, the target subspace is a subspace different from the error content.
[0115] In this embodiment, by writing the re-read algorithm model instructions and model parameters to a target subspace different from the subspace in which the error content is located, it is possible to avoid re-occurrence of an error due to hardware failure in the subspace in which the error content was located, provide accurate and effective algorithm model instructions for subsequent image processing, and further extend the fault-tolerant time and improve the safety of vehicle driving.
[0116] FIG. 9 is a flowchart of an image processing method by an artificial intelligence accelerator according to a further exemplary embodiment of the present disclosure.
[0117] In some optional embodiments, step 203 of reading the algorithm model instructions from a first pre-defined memory space that stores the algorithm model instructions by the artificial intelligence accelerator, executing the algorithm model instructions, and obtaining a processing result for the image to be processed includes the following steps 203a to 203b.
[0118] In step 203a, the artificial intelligence accelerator reads the algorithm model instruction from the first pre-defined memory space in which the algorithm model instruction is stored, and executes the algorithm model instruction to obtain an output result of the artificial intelligence accelerator.
[0119] The specific operation of this step can be referred to the above-mentioned embodiment, and the description is omitted here.
[0120] In step 203b, error results in the output results are filtered based on preset filtering rules to obtain a processing result for the processing target image.
[0121] Here, the preset filtering rule may be set according to actual needs, for example, the preset filtering rule may be a rule such as a multi-frame check or a Kalman filter, and is not specifically limited.
[0122] In this embodiment, after obtaining the output result of the accelerator, inference errors accidentally occurring in the output result are further filtered based on preset filtering rules, thereby further improving security.
[0123] The above embodiments of the present disclosure may be implemented alone or in any combination as long as they are not contradictory, and may be specifically configured according to actual needs, and the present disclosure is not limited thereto.
[0124] Any image processing method by an artificial intelligence accelerator according to the embodiments of the present disclosure may be executed by any suitable device having data processing capability, including but not limited to a terminal device and a server. Alternatively, any image processing method by an artificial intelligence accelerator according to the embodiments of the present disclosure may be executed by a processor, for example, the processor executes any image processing method by an artificial intelligence accelerator according to the embodiments of the present disclosure by calling corresponding instructions stored in a memory. The following description is omitted.
[0125] (Exemplary Apparatus) 10 is a structural schematic diagram of an image processing apparatus with an artificial intelligence accelerator according to an exemplary embodiment of the present disclosure. The apparatus of this embodiment is for implementing the corresponding embodiment of the image processing method with an artificial intelligence accelerator of the present disclosure. The apparatus shown in FIG. 10 includes a first control module 51, a second control module 52 and an artificial intelligence accelerator 53.
[0126] The first control module 51 is for determining algorithmic model information corresponding to the image to be processed.
[0127] The second control module 52 is for determining, based on the algorithm model information, an artificial intelligence accelerator that needs to execute an algorithm model instruction corresponding to the algorithm model information.
[0128] The artificial intelligence accelerator 53 is for reading the algorithm model instruction from a first pre-defined memory space storing the algorithm model instruction, and executing the algorithm model instruction to obtain a processing result for the processing target image.
[0129] FIG. 11 is a structural schematic diagram of an apparatus for processing an image by an artificial intelligence accelerator according to another exemplary embodiment of the present disclosure.
[0130] In some alternative embodiments, the first control module 51 includes a first determination unit 511 and a second determination unit 512 .
[0131] The first determining unit 511 is for determining the current frame information corresponding to the image to be processed.
[0132] The second determining unit 512 is for determining, according to a mapping rule between frame information and algorithm model information, the algorithm model information corresponding to the current frame information from the plurality of algorithm model information.
[0133] Here, the multiple pieces of algorithm model information each correspond to the same algorithm model instruction in a different storage sub-space in the first preset storage space.
[0134] In some alternative embodiments, the device of the present disclosure further comprises a reading module 61 and a writing module 62 .
[0135] The reading module 61 is for reading, based on the preset information, the algorithm model instruction required for image processing from the second preset storage space.
[0136] The write module 62 is for writing the algorithmic model instructions into a plurality of different storage sub-spaces of the first pre-defined storage space.
[0137] In some alternative embodiments, the second control module 52 includes a third determination unit 521 and a fourth determination unit 522 .
[0138] The third determining unit 521 is for determining an alternating scheduling rule of the accelerator corresponding to the algorithm model information.
[0139] The fourth determination unit 522 is for determining, based on the accelerator alternating scheduling rule, from the multiple artificial intelligence accelerators, an artificial intelligence accelerator that needs to execute the algorithm model instruction corresponding to the current algorithm model information.
[0140] In some alternative embodiments, the fourth determining unit 522 specifically comprises: The accelerator alternation order is determined based on the accelerator alternation scheduling rule, and the artificial intelligence accelerator that needs to execute the algorithm model instruction corresponding to the current algorithm model information according to the accelerator alternation order is determined.
[0141] In some alternative embodiments, the fourth determining unit 522 specifically comprises: The method includes: determining identification information corresponding to multiple algorithm models based on algorithm model information; determining an accelerator alternating order based on accelerator alternating scheduling rules; determining multiple algorithm model groups including identification information of at least one algorithm model corresponding to the algorithm model information based on the identification information corresponding to each algorithm model and a predetermined grouping rule; and for any algorithm model group, determining an artificial intelligence accelerator that currently needs to execute an algorithm model instruction corresponding to the algorithm model group according to the accelerator alternating order based on the historical accelerator scheduling information corresponding to the algorithm model group.
[0142] In some optional embodiments, an artificial intelligence accelerator that needs to execute an algorithm model instruction corresponding to any algorithm model group is configured to read the algorithm model instruction corresponding to the algorithm model group from a first pre-set memory space, and execute the algorithm model instruction to obtain a processing result corresponding to the algorithm model group.
[0143] The apparatus of the present disclosure further comprises a third control module 54 for determining a processing result for the processing target image based on the processing results corresponding to each algorithmic model group respectively.
[0144] In some optional embodiments, referring to FIG. 11, the device of the present disclosure further includes a reading module 61 and a writing module 62 .
[0145] The reading module 61 is for reading, based on the preset information, the algorithm model instruction required for image processing from the second preset storage space.
[0146] The write module 62 is for writing the algorithm model instructions into the first pre-defined storage space.
[0147] In some alternative embodiments, the apparatus of the present disclosure further comprises a first pre-defined memory space 63 for storing a plurality of algorithmic model instructions and model parameters corresponding to each algorithmic model instruction.
[0148] In some alternative embodiments, the first pre-defined storage space 63 can be a storage space in a memory.
[0149] In some alternative embodiments, the apparatus of the present disclosure further comprises a check module 64 and a repair module 65 .
[0150] The checking module 64 is for checking the preset contents of the algorithm model instructions of the plurality of parts of the first preset memory space and the model parameters corresponding to each algorithm model instruction respectively according to a preset period, and obtaining a checking result.
[0151] The repair module 65 is for performing a repair process on the preset content based on the preset repair rule in response to the check result to obtain a repair result.
[0152] In some alternative embodiments, the repair module 65 specifically: and in response to the checking result, delete the algorithm model instructions and model parameters in which the error content exists from the first preset storage space, re-read the algorithm model instructions and model parameters corresponding to the deleted content from the second preset storage space, and write the re-read algorithm model instructions and model parameters to a target sub-space in the first preset storage space, where the target sub-space is a sub-space different from the error content.
[0153] In some optional embodiments, the artificial intelligence accelerator 53 is specifically for reading algorithm model instructions from a first pre-set memory space that stores algorithm model instructions, and executing the algorithm model instructions to obtain an output result of the artificial intelligence accelerator.
[0154] The third control module 54 is further for filtering the error result in the output result according to a preset filtering rule to obtain a processing result for the processing target image.
[0155] In some alternative embodiments, the device of the present disclosure further comprises a second pre-defined storage space 66 for storing algorithmic model instructions. The second pre-defined storage space 66 can be a storage space in an external memory such as a hard disk, a floppy disk, an optical disk, etc.
[0156] The beneficial technical effects corresponding to the exemplary embodiments of the present apparatus may refer to the corresponding beneficial technical effects of the above exemplary method parts, and will not be described here.
[0157] 12 is a structural schematic diagram of an AI chip according to an exemplary embodiment of the present disclosure, the AI chip includes a first pre-defined memory space and an AI accelerator-based image processing device, the AI accelerator-based image processing device is for executing the AI accelerator-based image processing method according to any of the above embodiments.
[0158] In some optional embodiments, the artificial intelligence chip of the present disclosure may further include other related modules or units, such as hardware modules such as an internal bus, other IP (Intellectual Property) cores, CPU (Central Processing Unit) clusters, safety zones, and software modules such as post-processing modules, and may be specifically configured according to actual needs.
[0159] The beneficial effects corresponding to this chip embodiment can be referred to the corresponding beneficial technical effects of the above exemplary method parts, and the description thereof will be omitted here.
[0160] (Example Electronic Devices) FIG. 13 is a structural diagram of an electronic device according to an embodiment of the present disclosure. The electronic device includes one or more processors 11 and a memory 12.
[0161] The processor 11 may be a central processing unit (CPU) or other type of processing unit having data processing and / or instruction execution capabilities, and may control other components of the electronic device 10 to perform desired functions.
[0162] The memory 12 may include one or more computer program products, which may include various forms of computer-readable storage media, such as, for example, volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache). The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored in the computer-readable storage media, and the processor 11 may execute the one or more computer program instructions to realize the methods of the embodiments of the present disclosure described above and / or other desired functions.
[0163] In one example, electronic device 10 may further include input device(s) 13 and output device(s) 14 connected to each other via a bus system and / or other form of connection (not shown).
[0164] The input device 13 may further include, for example, a keyboard, a mouse, and the like.
[0165] The output device 14 can output various information to the outside, and can include, for example, a display, a speaker, a printer, a communication network, and a remote output device connected thereto.
[0166] For simplicity, Fig. 13 shows only some of the components related to the present disclosure in the electronic device 10, and omits components such as a bus, an input / output interface, etc. In addition, the electronic device 10 may further include any other appropriate components according to a specific application situation.
[0167] Exemplary Computer Program Products and Computer-Readable Storage Media In addition to the above methods and apparatus, embodiments of the present disclosure may further provide a computer program product including computer program instructions that, when executed by a processor, cause the processor to perform the steps of the methods of each type of embodiment of the present disclosure described in the "Exemplary Methods" section above.
[0168] The computer program product may be produced with program code for carrying out operations of embodiments of the present disclosure in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed as follows: entirely on the user computing device, partially on the user device, as a separate software package, partly on the user computing device and partly on a remote computing device, or entirely on the remote computing device or a server.
[0169] Moreover, the embodiments of the present disclosure may further provide a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, cause the processor to perform the steps of the methods of each type of embodiment of the present disclosure described in the "Exemplary Method" section above.
[0170] The computer readable storage medium may be any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may include, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.
[0171] Although the basic principles of the present disclosure have been described above with reference to specific examples, the advantages, merits, effects, etc. mentioned in the present disclosure are merely illustrative and not limiting, and these advantages, merits, effects, etc. are not necessarily possessed by each embodiment of the present disclosure. In addition, the specific details of the above disclosure are merely illustrative and easy-to-understand actions and are not limiting, and the above details do not necessarily limit the present disclosure to be realized by the above specific details.
[0172] Those skilled in the art can make various modifications and variations to the present disclosure without departing from the spirit and scope of the present disclosure. Thus, if these modifications and variations of the present disclosure fall within the scope of the claims of the present disclosure and their equivalents, the present disclosure is also intended to include these modifications and variations.
Claims
1. determining algorithm model information corresponding to the image to be processed; determining an artificial intelligence accelerator that needs to execute an algorithm model instruction corresponding to the algorithm model information according to the algorithm model information; and a step of reading, by the artificial intelligence accelerator, the algorithm model instruction from a first pre-defined memory space in which the algorithm model instruction is stored, and executing the algorithm model instruction to obtain a processing result for the image to be processed.
2. The step of determining algorithm model information corresponding to the image to be processed includes: determining current frame information corresponding to the image to be processed; 2. The image processing method using an artificial intelligence accelerator according to claim 1, further comprising: determining algorithm model information corresponding to the current frame information from a plurality of algorithm model information according to a mapping rule between frame information and algorithm model information, wherein the plurality of algorithm model information respectively correspond to the same algorithm model instruction in different memory sub-spaces in the first preset memory space.
3. Prior to the step of determining algorithm model information corresponding to the image to be processed, Reading an algorithm model command required for image processing from a second preset storage space according to preset information; 2. The method of claim 1, further comprising the step of: writing said algorithm model instructions into a plurality of different memory sub-spaces of said first pre-defined memory space.
4. The step of determining an artificial intelligence accelerator that needs to execute an algorithm model instruction corresponding to the algorithm model information according to the algorithm model information includes: determining an accelerator alternation scheduling rule corresponding to the algorithm model information; and determining an artificial intelligence accelerator that needs to execute an algorithm model instruction corresponding to the current algorithm model information from among the plurality of artificial intelligence accelerators based on an alternating scheduling rule of the accelerator.
5. According to the accelerator alternating scheduling rule, determining an AI accelerator that needs to execute an algorithm model instruction corresponding to the current algorithm model information from among the plurality of AI accelerators, determining an accelerator alternation order according to the accelerator alternation scheduling rule; and determining an artificial intelligence accelerator that currently needs to execute an algorithm model instruction corresponding to the algorithm model information based on the accelerator alternating order.
6. According to the accelerator alternating scheduling rule, determining an AI accelerator that currently needs to execute an algorithm model instruction corresponding to the algorithm model information from among a plurality of AI accelerators, determining identification information corresponding to a plurality of algorithm models respectively based on the algorithm model information; determining an accelerator alternation order according to the accelerator alternation scheduling rule; A step of determining a plurality of algorithm model groups corresponding to the algorithm model information based on the identification information corresponding to each of the algorithm models and a predetermined grouping rule, each of the algorithm model groups including identification information of at least one algorithm model; The image processing method using an artificial intelligence accelerator according to claim 4, further comprising: for any of the algorithm model groups, determining, according to the accelerator alternating order, the artificial intelligence accelerator that currently needs to execute the algorithm model instructions corresponding to the algorithm model group based on the historical accelerator scheduling information corresponding to the algorithm model group.
7. The step of reading the algorithm model instruction from a first preset memory space in which the algorithm model instruction is stored by the artificial intelligence accelerator and executing the algorithm model instruction to obtain a processing result for the processing target image includes: For an artificial intelligence accelerator that needs to execute an algorithm model instruction corresponding to any of the algorithm model groups, read the algorithm model instruction corresponding to the algorithm model group from the first preset memory space by the artificial intelligence accelerator, and execute the algorithm model instruction to obtain a processing result corresponding to the algorithm model group; and determining a processing result for the processing target image based on the processing results corresponding to each of the algorithm model groups, respectively.
8. Prior to the step of determining algorithm model information corresponding to the image to be processed, Reading an algorithm model command required for image processing from a second preset storage space according to preset information; 2. The method for image processing by an artificial intelligence accelerator according to claim 1, further comprising: writing said algorithm model instructions into said first pre-defined memory space.
9. The first preset storage space stores a plurality of the algorithm model instructions and model parameters corresponding to each of the algorithm model instructions; The image processing method by the artificial intelligence accelerator includes: According to a preset period, respectively checking the preset contents in the plurality of parts of the algorithm model instructions in the first preset storage space and model parameters corresponding to each of the algorithm model instructions to obtain a checking result; 2. The image processing method using an artificial intelligence accelerator as claimed in claim 1, further comprising: in response to the check result, performing a restoration process on the pre-defined content based on a pre-defined restoration rule to obtain a restoration result.
10. performing a repair process on the preset content based on a preset repair rule to obtain a repair result, Deleting the algorithm model instructions and model parameters in which error content exists from the first preset storage space, and re-reading the algorithm model instructions and model parameters corresponding to the deleted content from the second preset storage space; 10. The image processing method by the artificial intelligence accelerator of claim 9, further comprising: writing the re-read algorithm model instructions and model parameters to a target subspace in the first preset memory space, the target subspace being a subspace different from the error content subspace.
11. The step of reading the algorithm model instruction from a first preset memory space in which the algorithm model instruction is stored by the artificial intelligence accelerator and executing the algorithm model instruction to obtain a processing result for the processing target image includes: reading the algorithm model instruction from a first preset memory space in which the algorithm model instruction is stored, and executing the algorithm model instruction by the artificial intelligence accelerator to obtain an output result of the artificial intelligence accelerator; 2. The image processing method using an artificial intelligence accelerator according to claim 1, further comprising: a step of filtering error results in the output result based on preset filtering rules to obtain a processing result for the processing target image.
12. a first control module for determining algorithmic model information corresponding to the target image; A second control module for determining an artificial intelligence accelerator that needs to execute an algorithm model instruction corresponding to the algorithm model information according to the algorithm model information; and an artificial intelligence accelerator for reading the algorithm model instructions from a first pre-defined memory space in which the algorithm model instructions are stored, and executing the algorithm model instructions to obtain a processing result for the image to be processed.
13. A first preset memory space and an image processing device with an artificial intelligence accelerator; An artificial intelligence chip, characterized in that the image processing device using an artificial intelligence accelerator is for executing the image processing method using an artificial intelligence accelerator according to any one of claims 1 to 11.
Citation Information
Patent Citations
High dynamic range image processing with fixed calibration settings
JP2022173094A
Systems and method for data survivability
US8234461B2
Memory access and detecting memory failures using dynamically replicated memory based on a replication policy
US9916116B2
