Abnormal link detection method and device, electronic equipment and storage medium

Abstract

The present disclosure provides an abnormal link detection method and device, electronic equipment and storage medium, relates to the field of image processing, in particular to the field of automatic driving, high-precision map, intelligent transportation and computer image processing. The specific implementation scheme is: acquiring an automatic driving image, wherein the automatic driving image is an image generated in an automatic driving process; in a case where it is detected that the automatic driving image is an abnormal image, determining an abnormal image processing link from an image processing full link according to detection information of at least one image processing link included in the image processing full link, wherein the image processing link is used for processing the image generated in the automatic driving process, and the abnormal image processing link represents the image processing link generating the abnormal image.

Description

Technical Field

[0001] This disclosure relates to the field of image processing, and more particularly to the fields of autonomous driving, high-precision mapping, intelligent transportation, and computer image processing. Specifically, it relates to an abnormal link detection method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the continuous development of autonomous driving technology, the demand for image processing during the autonomous driving process is increasing. For example, for abnormal images generated during autonomous driving, it is necessary to detect the source of the anomaly. Summary of the Invention

[0003] This disclosure provides an abnormal link detection method, apparatus, electronic device, and storage medium.

[0004] According to one aspect of this disclosure, an abnormal link detection method is provided, comprising: acquiring an autonomous driving image, wherein the autonomous driving image is an image generated during autonomous driving; and, when the autonomous driving image is detected to be an abnormal image, determining an abnormal image processing link from the image processing full link based on detection information of at least one image processing link included in the image processing full link, wherein the image processing link is used to process the image generated during the autonomous driving process, and the abnormal image processing link characterizes the image processing link that generated the abnormal image.

[0005] According to another aspect of this disclosure, an abnormal link detection apparatus is provided, comprising: an acquisition module for acquiring an autonomous driving image, wherein the autonomous driving image is an image generated during autonomous driving; and an anomaly detection module for, when detecting that the autonomous driving image is an abnormal image, determining an abnormal image processing link from the entire image processing link based on detection information of at least one image processing link included in the entire image processing link, wherein the image processing link is used to process the image generated during the autonomous driving process, and the abnormal image processing link characterizes the image processing link that generated the abnormal image.

[0006] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method as described in this disclosure.

[0007] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform the methods described in this disclosure.

[0008] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the methods described in this disclosure.

[0009] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0010] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0011] Figure 1 This illustration schematically shows an exemplary system architecture of an abnormal link detection method and apparatus according to embodiments of the present disclosure;

[0012] Figure 2 A flowchart illustrating an abnormal link detection method according to an embodiment of the present disclosure is shown schematically.

[0013] Figure 3 This illustration schematically shows a flowchart of detecting whether the current image processing link is an abnormal image processing link according to an embodiment of the present disclosure;

[0014] Figure 4 This diagram illustrates an application scenario of image processing according to an embodiment of the present disclosure.

[0015] Figure 5A A schematic diagram illustrating the frame rate of the first integrity information according to an embodiment of the present disclosure is shown.

[0016] Figure 5B A schematic diagram illustrating the frame rate of the second integrity information according to an embodiment of the present disclosure is shown.

[0017] Figure 5C A schematic diagram illustrating the frame rate of third integrity information according to an embodiment of the present disclosure is shown.

[0018] Figure 6 A block diagram of an abnormal link detection apparatus according to an embodiment of the present disclosure is schematically shown; and

[0019] Figure 7 A block diagram of an electronic device suitable for implementing an abnormal link detection method according to an embodiment of the present disclosure is shown schematically. Detailed Implementation

[0020] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0021] Abnormal autonomous driving images may appear during autonomous driving operations. Due to the complexity of the autonomous driving image processing chain, which involves numerous hardware and software modules, it is difficult to accurately locate the abnormal image processing chain that generates the abnormal images.

[0022] To address this, this disclosure proposes an abnormal link detection scheme. For example, it acquires an autonomous driving image, wherein the autonomous driving image is an image generated during the autonomous driving process; and when an abnormal image is detected, it determines an abnormal image processing link from the entire image processing chain based on detection information of at least one image processing link included in the entire image processing chain, wherein the image processing link is used to process the image generated during the autonomous driving process, and the abnormal image processing link characterizes the image processing link that generated the abnormal image.

[0023] According to embodiments of this disclosure, by acquiring autonomous driving images, and in the case where the autonomous driving images are detected as abnormal images, the detection information of at least one image processing link within the entire image processing chain is processed to determine the abnormal image processing link from the entire image processing chain. This not only realizes the detection of abnormal links in the entire image processing chain, but also realizes the source localization of abnormal image processing links, thereby improving the localization accuracy and detection efficiency of abnormal links.

[0024] Figure 1 The illustration schematically shows an exemplary system architecture of the abnormal link detection method and apparatus according to embodiments of the present disclosure.

[0025] It is important to note that Figure 1 The examples shown are merely examples of system architectures that can be applied to the embodiments of this disclosure, in order to help those skilled in the art understand the technical content of this disclosure, but do not mean that the embodiments of this disclosure cannot be used in other devices, systems, environments or scenarios.

[0026] like Figure 1 As shown, the system architecture 100 according to this embodiment may include a vehicle 101, a vision sensor 102, a network 103, and an electronic device 104. The network 103 serves as a medium for providing a communication link between the vehicle 101 and the electronic device 104. The network 103 may include various connection types, such as at least one of wired and wireless communication links.

[0027] Vehicle 101 may include internal combustion engine powered vehicles, electric vehicles, or hybrid electric vehicles. For example, vehicle 101 may be a vehicle equipped with an automatic control system. Vehicle 101 may be an autonomous driving vehicle. Vehicle 101 may be equipped with data collection devices to gather information about the surrounding environment.

[0028] The vision sensor 102 is used to acquire images of the vehicle 101 during autonomous driving. The vision sensor 102 can be mounted on the vehicle 101, for example, on the outer top of the vehicle 101, or inside the vehicle 101.

[0029] The vision sensor 102 can be used to acquire images of objects while the vehicle 101 is in motion. The vision sensor 102 can be a camera of various types.

[0030] Furthermore, the vision sensor 102 can be integrated with the vehicle 101 or it can be separate from the vehicle 101; no limitation is made here.

[0031] Electronic device 104 may include at least one of a terminal device and a server. The electronic device may include a central processing unit (CPU) and a graphics processing unit (GPU). The CPU and GPU may communicate via a PCIe (Peripheral Component Interconnect Express) link. The CPU and vision sensor 102 may also communicate via a PCIe link. The terminal device may be various electronic devices with a display screen and supporting web browsing. For example, the terminal device may include at least one of a smartphone, tablet computer, laptop computer, and desktop computer.

[0032] A server can be a server that provides various services. For example, a server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system. It solves the shortcomings of traditional physical hosts and VPS (Virtual Private Server) services, such as high management difficulty and weak business scalability.

[0033] For example, electronic device 104 can be used to execute the abnormal link detection method of the present disclosure embodiments. Electronic device 104 can receive autonomous driving images from vehicle 101. When electronic device 104 detects that the autonomous driving image is an abnormal image, it determines the abnormal image processing link from the entire image processing link based on the detection information of at least one image processing link included in the entire image processing link, wherein the image processing link is used to process images generated during autonomous driving, and the abnormal image processing link represents the image processing link that generated the abnormal image.

[0034] For example, electronic device 104 can also receive autonomous driving images from vision sensor 102 so that electronic device 104 can perform the above-described abnormal link detection method.

[0035] The abnormal link detection method provided in this disclosure can generally also be executed by the electronic device 104. Correspondingly, the abnormal link detection device provided in this disclosure can generally be disposed in the electronic device 104. The abnormal link detection method provided in this disclosure can also be executed by a server or server cluster that is different from the electronic device 104 and capable of communicating with at least one of the vehicle 101 and the electronic device 104. Correspondingly, the abnormal link detection device provided in this disclosure can also be disposed in a server or server cluster that is different from the electronic device 104 and capable of communicating with at least one of the vehicle 101 and the electronic device 104.

[0036] It should be noted that the abnormal link detection method provided in this embodiment can also be executed by the vehicle 101. Accordingly, the abnormal link detection device provided in this embodiment can also be installed in the vehicle 101.

[0037] It should be understood that Figure 1 The number of vehicles, vision sensors, networks, and servers shown is merely illustrative. Any number of vehicles, vision sensors, networks, and servers can be included depending on implementation needs.

[0038] It should be noted that the sequence numbers of the operations in the following methods are for descriptive purposes only and should not be considered as indicating the execution order of the operations. Unless explicitly stated otherwise, the method does not need to be executed in the exact order shown.

[0039] Figure 2 A flowchart illustrating an abnormal link detection method according to an embodiment of the present disclosure is shown schematically.

[0040] like Figure 2 As shown, the method 200 includes operations S210~S220.

[0041] During operation of S210, images of autonomous driving are acquired.

[0042] In operation S220, if the autonomous driving image is detected to be an abnormal image, the abnormal image processing link is determined from the entire image processing link based on the detection information of at least one image processing link included in the entire image processing link.

[0043] According to embodiments of this disclosure, the autonomous driving image can be an image generated during the autonomous driving process. The autonomous driving image can be a map image generated by the vehicle during autonomous driving. The map image can include at least one of the following: an actual road image, a sign image, etc. The autonomous driving image can also be an image of the actual traffic conditions ahead of the vehicle during the autonomous driving process; for example, the actual traffic conditions image can include location information and distance information of vehicles ahead from the autonomous driving vehicle.

[0044] According to embodiments of this disclosure, during autonomous driving, the original images acquired by the vision sensor are processed to generate the final autonomous driving image. However, during the generation of the autonomous driving image, abnormal situations such as frame loss and incomplete image information may occur, resulting in an abnormal autonomous driving image.

[0045] According to embodiments of this disclosure, after acquiring an autonomous driving image, it can be determined whether the autonomous driving image is an abnormal image based on the actual situation.

[0046] For example, when the autonomous driving image includes a map image, the autonomous driving image is determined to be an abnormal image in response to detecting at least one of the following: low image clarity, incomplete map image, lack of sign information, and lack of road label information.

[0047] For example, when the autonomous driving image includes an image of the actual vehicle condition, in response to detecting at least one of the following in the autonomous driving image: low image clarity, image sharpening, abnormal contrast display, incomplete position and distance information of obstacles and autonomous vehicles, the autonomous driving image is determined to be an abnormal image.

[0048] The above-described process for determining abnormal images is only an exemplary embodiment, but is not limited thereto, and may also include other abnormal images known in the field of autonomous driving.

[0049] According to embodiments of this disclosure, an image processing link is used to process images generated during autonomous driving, and an abnormal image processing link characterizes the image processing link that generates abnormal images.

[0050] According to embodiments of this disclosure, a complete image processing chain is used to process autonomous driving images. Since the processing of autonomous driving images is complex and involves numerous hardware or software modules, the image processing process can be divided into multiple image processing chains based on the image processing functions, collectively forming the complete image processing chain.

[0051] According to embodiments of this disclosure, the entire image processing chain may include multiple image processing links. For any image processing link in the entire image processing chain, based on the detection information of any image processing link, it can be determined whether the image processing link is an abnormal processing link.

[0052] According to embodiments of this disclosure, the detection information includes at least one of the following: integrity information, transmission link status information, and cache status information. Integrity information characterizes the integrity of the image input to the image processing link; for example, integrity information may include at least one of the following: whether frame drops exist, the time of frame drops, and the number of frame drops. Transmission link status information characterizes the transmission link status of the image processing link; for example, the transmission link status between multiple sensors or multiple processors within the image processing link. Cache status information characterizes the storage resource usage of the cache within the image processing link.

[0053] According to embodiments of this disclosure, the detection information for multiple image processing links can be different. The detection information for an image processing link can simultaneously include integrity information, transmission link status information, and buffer status information. The detection information can also include any two of the following: integrity information, transmission link status information, and buffer status information. Alternatively, the detection information can include only one of the following: integrity information, transmission link status information, and buffer status information.

[0054] For example, the detection information for an image processing link can include integrity information, transmission link status information, and buffer status information.

[0055] It should be noted that the detection information of the image processing link can be set according to the actual situation, and the types of information within the detection information can also be set according to the actual situation, so as to ensure that the detection information can be used to determine whether the image processing link is a normal image processing link.

[0056] It should be understood that the division principles between multiple image processing links, the number of image processing links, the image processing order, and other information can all be determined according to the actual situation. The embodiments disclosed herein are only exemplary embodiments, and the specific image processing links are not limited thereto.

[0057] According to embodiments of this disclosure, by acquiring autonomous driving images, and in the case where the autonomous driving images are detected as abnormal images, the detection information of at least one image processing link within the entire image processing chain is processed to determine the abnormal image processing link from the entire image processing chain. This not only realizes the detection of abnormal links in the entire image processing chain, but also realizes the source localization of abnormal image processing links, thereby improving the localization accuracy and detection efficiency of abnormal links.

[0058] According to embodiments of this disclosure, operation S220 may include repeatedly performing the following operations until an abnormal image processing link is identified from the entire image processing link.

[0059] Determine the current detection information of the current image processing link. If the current image processing link is determined to be a normal image processing link based on the current detection information, then the next image processing link is determined as the new current image processing link.

[0060] According to embodiments of this disclosure, the current image processing link can represent the current level of the image processing link in the entire image processing link. The next image processing link can be the next level of the current image processing link.

[0061] According to embodiments of this disclosure, the entire image processing chain may include multiple image processing chains. These multiple image processing chains may correspond to multiple levels of image processing chains, and there is a sequential relationship between the execution order of these multiple levels of image processing chains.

[0062] Because the entire image processing chain consists of multiple levels, the process of identifying abnormal image processing chains involves determining whether the current image processing chain is a normal one based on the detection information of that chain. If the current chain is determined to be normal, then the process proceeds to determine whether the next level of image processing chain is also normal.

[0063] For example, since multiple image processing links correspond to multiple detection information, in determining whether the current image processing link is a normal image processing link, the current detection information of the current image processing link is first determined, and then the current image processing link is determined as a normal image processing link based on the current detection information.

[0064] If the current image processing link is determined to be a normal image processing link, the next level image processing link is determined as the new current image processing link. Similarly, the process of determining whether the current image processing link is a normal image processing link is repeated, so that all image processing links in the entire image processing link can be traversed and abnormal image processing links can be identified from the entire image processing link.

[0065] According to embodiments of this disclosure, the entire image processing link may include a first image processing link, which characterizes the vision sensor image processing link. When the current image processing link is the first image processing link, the current detection information may include at least one of the following: first integrity information, first transmission link status information, and first buffer status information. The first integrity information characterizes the integrity of the original image input to the vision sensor, and the first transmission link status information characterizes the transmission link status of the vision sensor.

[0066] According to embodiments of this disclosure, the above-described abnormal link detection method may further include the following operations.

[0067] If at least one of the following conditions is met, namely, that the first image processing link does not have frame loss, that the first image processing link is a normal transmission link, or that the first buffer is not full, the first image processing link is determined to be a normal image processing link.

[0068] According to embodiments of this disclosure, it is determined whether the current image processing link is a normal image processing link based on current detection information. Since the current detection information may include at least one of the following: first integrity information, first transmission link status information, and first buffer status information, it is accordingly determined whether the current image processing link is a normal image processing link based on the current detection information.

[0069] For example, if the current detection information of the first image processing link only includes the first integrity information, the first image processing link can be determined to be a normal image processing link in response to determining that there are no dropped frames in the first image processing link based on the first integrity information. Similarly, if the current detection information of the first image processing link only includes the first transmission link status information or the first buffer status information, the first image processing link can be determined to be a normal image processing link in response to determining that the transmission link of the first image processing link is a normal transmission link based on the first transmission link status information or determining that the buffer is not full based on the first buffer status information.

[0070] For example, if the current detection information of the first image processing link includes first integrity information, first transmission link status information and first buffer status information, in response to any one of the detection information of the first integrity information, the first transmission link status information and the first buffer status information satisfying a first predetermined condition, the first image processing link can be determined to be a normal image processing link.

[0071] As another embodiment, if the first integrity information, the first transmission link status information, and the first buffer status information all satisfy the first predetermined condition, the first image processing link can be determined to be a normal image processing link. For example, if it is determined from the first integrity information that the first image processing link does not have frame loss, and from the first transmission link status information that the transmission link of the first image processing link is a normal transmission link, and from the first buffer status information that the buffer is not full, the first image processing link is determined to be a normal image processing link.

[0072] According to embodiments of this disclosure, first integrity information can characterize the integrity of the original image input to the visual sensor. The first integrity information may include at least one of the following: whether the original image input to the visual sensor contains dropped frames, the time of the dropped frames, the number of dropped frames, etc. The process of determining the first integrity information may include the following operations.

[0073] According to embodiments of this disclosure, a frame rate image acquired by a vision sensor is determined. First integrity information is determined based on the frame rate image. The frame rate image acquired by the vision sensor can represent the relationship between the frame rate and time of the original image. The first integrity information can be determined based on the continuity and stability of the frame rate in the frame rate image.

[0074] For example, if the frame rate of the original image in the frame rate image is stable at the same value and the frame rate is consistent throughout the acquisition time, it indicates that the first integrity information is that there are no dropped frames.

[0075] For example, the frame rate of the original image in the frame rate image fluctuates at time A, losing 3 frames, and there is a break in the frame rate between time A and time B, indicating that the first integrity information is that there are frame drops, the time of frame drops is between time A and time B, and the number of dropped frames is 3.

[0076] According to embodiments of this disclosure, after determining the first integrity information, it is possible to directly determine whether there is frame loss in the first image processing link based on the first integrity information.

[0077] According to the implementation of this disclosure, the above-mentioned abnormal link detection method also includes the following operations.

[0078] According to embodiments of this disclosure, a signal eye diagram of the transmission link from the vision sensor to the central processing unit is determined. Based on the signal eye diagram, first transmission link status information is obtained.

[0079] According to embodiments of this disclosure, an eye diagram is a graph observed on an oscilloscope when the performance of a transmission system is estimated experimentally and improved through adjustments. For example, an eye diagram can be formed by superimposing the waveforms of each symbol obtained from a scan, based on the persistence effect of the oscilloscope.

[0080] The signal eye diagram contains a wealth of information that can reflect the overall characteristics of a digital signal, thereby assessing the transmission quality of the digital signal. For example, the effects of intersymbol interference and noise can be observed from the signal eye diagram, thus estimating the quality of the signal transmission state. Therefore, the transmission link state information from the vision sensor to the central processing unit can be determined based on the signal eye diagram.

[0081] According to embodiments of this disclosure, when the current image processing link is the first image processing link, a signal eye diagram of the transmission link from the vision sensor to the central processing unit can be generated by an oscilloscope.

[0082] According to embodiments of this disclosure, after determining the signal eye diagram of the transmission link from the vision sensor to the central processing unit, first transmission link status information can be obtained based on whether the signal eye diagram conforms to a standard protocol.

[0083] For example, based on the data transmission type from the vision sensor to the central processing unit, a standard protocol corresponding to the aforementioned transmission type is obtained. Eye diagram data conforming to the aforementioned standard protocol is acquired from the signal eye diagram, and this eye diagram data is compared with the standard data defined by the standard protocol to obtain comparison information. If the comparison information indicates that the eye diagram data meets the standard data, the first transmission link status information is determined to meet the standard. Therefore, based on the first transmission link status information, it can be determined that the transmission link of the first image processing link is a normal transmission link.

[0084] According to embodiments of this disclosure, eye diagram data may include at least one of the following: the amplitude of the voltage signal, the duration of the voltage signal, etc.

[0085] For example, standard data can have predetermined amplitude thresholds and predetermined duration thresholds. The process of comparing the above eye diagram data with the standard data defined by the standard protocol is as follows: determine whether the amplitude of the voltage signal is greater than the predetermined amplitude threshold, and determine whether the duration of the voltage signal is greater than the predetermined duration threshold.

[0086] According to embodiments of this disclosure, the predetermined amplitude threshold or predetermined duration threshold can be determined based on the data transmission type or chip requirements.

[0087] According to embodiments of this disclosure, the data transmission type of the transmission link from the vision sensor to the central processing unit includes at least one of the following: Mobile Industry Processor Interface (MIPI) and Gigabit Multimedia Serial Link (GMSL).

[0088] According to embodiments of this disclosure, when it is determined that the transmission link from the vision sensor to the central processing unit includes a MIPI signal, a standard protocol and a signal eye diagram corresponding to the MIPI signal are acquired. The signal eye diagram of the MIPI signal is compared with the standard protocol, and if the signal eye diagram of the MIPI signal satisfies the standard protocol, it is determined that the transmission link of the first image processing link is a normal transmission link.

[0089] According to embodiments of this disclosure, when it is determined that the transmission link from the vision sensor to the central processing unit includes a GMSL signal, a standard protocol and a signal eye diagram corresponding to the GMSL signal are acquired. The signal eye diagram of the GMSL signal is compared with the standard protocol, and if the signal eye diagram of the GMSL signal satisfies the standard protocol, it is determined that the transmission link of the first image processing link is a normal transmission link.

[0090] If it is determined that the transmission link from the vision sensor to the central processing unit includes MIPI signals and GMSL signals, and if it is determined that the signal eye diagrams of both GMSL signals and MIPI signals meet the corresponding standard protocols, then the transmission link of the first image processing link is determined to be a normal transmission link.

[0091] According to embodiments of this disclosure, the first cache state information is used to characterize the storage resource usage of the visual sensor's cache. The first cache state information can be obtained in real time by setting a side thread.

[0092] For example, if the available storage resource value is determined to be less than or equal to a predetermined available storage resource threshold, the first cache status information can be cache status information indicating that the cache is full. If the available storage resource value is determined to be greater than the predetermined available storage resource threshold, the first cache status information can be cache status information indicating that the cache is not full.

[0093] According to embodiments of this disclosure, the entire image processing link may further include a second image processing link. The second image processing link may characterize a central processing unit (CPU) image processing link. The second image processing link may be the next level image processing link after the first image processing link.

[0094] According to embodiments of this disclosure, when the current image processing link is a second image processing link, the current detection information may include at least one of the following: second integrity information and second cache state information. The second integrity information may characterize the integrity of the first intermediate image input to the central processing unit. The first intermediate image may be obtained from the original image.

[0095] According to embodiments of this disclosure, the above-described abnormal link detection method may further include the following operations.

[0096] If at least one of the following conditions is met—that the second image processing link does not have frame loss based on the second integrity information and that the cache is not full based on the second cache status information—the second image processing link is determined to be a normal image processing link.

[0097] According to embodiments of this disclosure, the entire image processing link may further include a third image processing link, which may characterize a graphics processor image processing link. The third image processing link may be the next level image processing link after the second image processing link.

[0098] According to embodiments of this disclosure, when the current image processing link is a third image processing link, the current detection information may include at least one of the following: third integrity information, second transmission link status information, and third buffer status information. The third integrity information may characterize the integrity of the second intermediate image input to the graphics processor. The second intermediate image may be obtained based on the first intermediate image. The second transmission link status information may characterize the transmission link status of the graphics processor.

[0099] According to embodiments of this disclosure, the above-described abnormal link detection method may further include the following operations.

[0100] If at least one of the following conditions is met, namely, that the third image processing link does not have frame loss, that the third image processing link is a normal transmission link, or that the third buffer is not full, the third image processing link is determined to be a normal image processing link.

[0101] According to embodiments of this disclosure, when it is determined that the first image processing link is a normal image processing link, the next level image processing link of the first image processing link, namely the second image processing link, is determined as the new current image processing link. Based on the current detection information corresponding to the second image processing link, it can be determined whether the second image processing link is a normal image processing link.

[0102] If the second image processing link is determined to be a normal image processing link, the next level image processing link, namely the third image processing link, is designated as the new current image processing link. Based on the current detection information corresponding to the third image processing link, it can be determined whether the third image processing link is a normal image processing link.

[0103] The following is for reference. Figure 3 Figure 5 illustrates the abnormal link detection method according to the embodiments of this disclosure in further detail with reference to specific examples.

[0104] Figure 3A flowchart illustrating a method for detecting whether the current image processing link is an abnormal image processing link according to an embodiment of the present disclosure is shown.

[0105] like Figure 3 As shown, the method 300 includes operations S301 to S305.

[0106] In operation S301, does at least one of the first integrity information, the first transmission link status information, and the first buffer status information satisfy the first predetermined condition? If yes, then operation S302 is executed; if no, then operation S305 is executed.

[0107] In operation S302, does at least one of the second integrity information and the second cached state information satisfy the second predetermined condition? If yes, then operation S303 is executed; if no, then operation S305 is executed.

[0108] In operation S303, at least one of the third integrity information, the second transmission link status information, and the third buffer status information satisfies the third predetermined condition? If yes, then operation S304 is executed; if yes, then operation S305 is executed.

[0109] When operating S304, other exceptions are output.

[0110] When operating S305, an abnormal image processing link is output.

[0111] According to embodiments of this disclosure, the first predetermined condition may include at least one of the following: determining that there are no frame drops in the first image processing link based on first integrity information, determining that the transmission link of the first image processing link is a normal transmission link based on first transmission link status information, and determining that the cache is not full based on first cache status information.

[0112] As an example, the first predetermined condition may include determining that there are no frame drops in the first image processing link based on the first integrity information, determining that the transmission link of the first image processing link is a normal transmission link based on the first transmission link status information, and determining that the cache is not full based on the first cache status information.

[0113] According to embodiments of this disclosure, the second predetermined condition may include at least one of determining, based on second integrity information, that there are no dropped frames in the second image processing link and determining, based on second cache status information, that the cache is not full.

[0114] As one embodiment, the second predetermined condition may include determining that there are no frame drops in the second image processing link based on the second integrity information, and determining that the cache is not full based on the second cache status information.

[0115] According to embodiments of this disclosure, the third predetermined condition may include at least one of the following: determining that there are no frame drops in the third image processing link based on third integrity information, determining that the transmission link of the third image processing link is a normal transmission link based on second transmission link status information, and determining that the cache is not full based on third cache status information.

[0116] As an example, the third predetermined condition may include the third integrity information determining that there are no frame drops in the third image processing link, the second transmission link status information determining that the transmission link of the third image processing link is a normal transmission link, and the third buffer status information determining that the buffer is not full.

[0117] According to the embodiments of this disclosure, since it is determined that the first image processing link, the second image processing link, and the third image processing link are all normal image processing links, that is, there are no abnormal image processing links in the entire image processing link, other abnormalities are output in operation S304.

[0118] As an example, after operation S304 outputs other abnormalities, it is possible to verify whether there are any abnormalities in the acquisition of the original image, the display of the autonomous vehicle, or the information display of the upper-layer application. That is, when it is determined that the entire image processing link is a normal image processing link, it is possible to detect whether there are any abnormalities in other devices outside the image processing link.

[0119] According to embodiments of this disclosure, after determining that the current image processing link is an abnormal image processing link, the abnormal image processing link is output in operation S305. For example, if operation S301 determines that the first image processing link is an abnormal image processing link, operation S305 outputs "The first image processing is an abnormal image processing link". If operation S302 determines that the second image processing link is an abnormal image processing link, operation S305 outputs "The second image processing is an abnormal image processing link". If operation S303 determines that the third image processing link is an abnormal image processing link, operation S305 outputs "The third image processing is an abnormal image processing link".

[0120] It should be noted that, since the current image processing link can be the first image processing link, the second image processing link, or the third image processing link, abnormal link detection can directly start from operation S301, executing operations S301~S305 in the execution order. Abnormal link detection can also start from operation S302, executing operations S302~S305 in the execution order. Abnormal link detection can also start from operation S303, executing operations S303~S305 in the execution order.

[0121] As another embodiment, since the current image processing link can start from an image processing link other than the first image processing link, after other abnormalities are output in operation S304, operation S301 can be entered to start from the first image processing link and execute operations S301~S305 to re-detect the entire image processing link.

[0122] According to embodiments of this disclosure, the entire image processing chain may include a first image processing chain, a second image processing chain, and a third image processing chain, which respectively characterize the vision sensor image processing chain, the central processing unit image processing chain, and the graphics processing unit image processing chain. The original image is input to the vision sensor, processed by the first image processing chain to obtain a first intermediate image; the first intermediate image is input to the central processing unit, processed by the second image processing chain to obtain a second intermediate image; the second intermediate image is input to the image processor, processed by the third image processing chain to obtain an autonomous driving image.

[0123] Figure 4 The diagram illustrates an application scenario of image processing via an image processing link according to an embodiment of the present disclosure.

[0124] like Figure 4 As shown, in 400, the entire image processing chain includes a first image processing chain 401, a second image processing chain 402, and a third image processing chain 403. The first image processing chain 401 may include a vision sensor 4011, a serializer 4012, a deserializer 4013, and an FPGA 4014. The second image processing chain 402 may include a CPU 4021. The third image processing chain 403 may include a GPU 4031.

[0125] The vision sensor 4011 can be used to acquire raw images, and then send the raw images to the serializer 4012 via a MIPI signal so that the serializer 4012 can perform SerDes conversion. SerDes is short for SERializer / DESerializer. After the serializer 4012 performs SerDes conversion, the converted raw image is obtained. The serializer 4012 can transmit the converted raw image to the deserializer 4013 via a GMSL signal. The deserializer 4013 can process the converted raw image to obtain a processed image. The processed image can be transmitted to the FPGA 4014 via a MIPI signal so that the FPGA 4014 can perform image processing on the processed image and output a first intermediate image.

[0126] Image processing performed by the FPGA4014 on the processed image can include at least one of frame rate conversion and image optimization. For example, image optimization can be performed using an Image Signal Processor (ISP). The FPGA4014 can also perform driving processing on the processed image for use in autonomous driving applications. For example, the processed image can be compressed, such as converting YUV422 to YUV420. In YUV, "Y" can represent "luminance" (Luma), "U" can represent "chrominance" (Chrominance), and "V" can represent "chroma" (Chroma).

[0127] After the FPGA4014 outputs the first intermediate image, it transmits the first intermediate image to the CPU4021 of the second image processing link 402 via the PCIe transmission link.

[0128] CPU 4021 can perform image processing on the first intermediate image through image processing routines, and perform driving or transmission processing through other processing routines, thereby outputting the second intermediate image. For example, CPU 4021 can transmit the second intermediate image to GPU 4031 of the third image processing link 403 through the PCIe transmission link.

[0129] The GPU4031's application processing routines are used in higher-level applications such as perception, fusion, and localization for autonomous driving image processing. The GPU4031 processes a second intermediate image using image processing routines to obtain the autonomous driving image. The GPU4031 can also manage its load and utilization through transfer routines.

[0130] According to embodiments of this disclosure, when the current image processing link is a second image processing link, the process of determining the second integrity information may include the following operations.

[0131] The second integrity information is obtained based on the first processing frame rate.

[0132] According to embodiments of this disclosure, the first processing frame rate can characterize the processing frame rate of an image processing routine in a central processing unit.

[0133] According to embodiments of this disclosure, when the current image processing link is a third image processing link, the process of determining third integrity information may include the following operations.

[0134] The third integrity information is obtained based on the second processing frame rate.

[0135] According to embodiments of this disclosure, the second processing frame rate characterizes the processing frame rate of the image processing routine in the graphics processor.

[0136] According to embodiments of this disclosure, after a first intermediate image is input into a central processing unit (CPU), the CPU can process the first intermediate image and output a second intermediate image. The CPU includes image processing routines and other processing routines.

[0137] Similarly, after the second intermediate image is input into the graphics processor, the GPU can process the second intermediate image and output the autonomous driving image. The GPU includes image processing routines and other processing routines.

[0138] For example, the image processing routines in a CPU include processing routines related to processing a first intermediate image, while other processing routines may be processing routines unrelated to the first intermediate image. The image processing routines in a GPU include processing routines related to processing a second intermediate image, while other processing routines may be processing routines unrelated to the second intermediate image. These other processing routines may be transfer routines, data computation routines, driver routines, etc.

[0139] According to embodiments of this disclosure, the processing frame rate of the image processing routine in the CPU can be extracted to obtain second integrity information. Third integrity information is obtained by extracting the processing frame rate of the image processing routine in the GPU.

[0140] According to embodiments of this disclosure, if it is determined that there is frame loss in the first image processing link based on the first integrity information, it is determined that there is an anomaly in the hardware within the first image processing link, such as an anomaly in the FPGA.

[0141] According to an embodiment of this disclosure, if frame loss is determined to exist in the second image processing link based on the second integrity information, the abnormal image processing link is determined to be the second image processing link, and the abnormal problem is at least one of system software and driver processing abnormalities.

[0142] According to embodiments of this disclosure, when it is determined that there is frame loss in the third image processing link based on the third integrity information, the abnormal image processing link is identified as the third image processing link, and the abnormal problem is at least one of system software and driver processing abnormalities.

[0143] Figure 5A A schematic diagram illustrating the frame rate of the first integrity information according to an embodiment of the present disclosure is shown. Figure 5B A schematic diagram illustrating the frame rate of the second integrity information according to an embodiment of the present disclosure is shown. Figure 5C A schematic diagram illustrating the frame rate of third integrity information according to an embodiment of the present disclosure is shown.

[0144] According to an embodiment of this disclosure, taking a hardware configuration of 10 200MP cameras, an Intel X86 Xeon E5 CPU, and 2 * RTX1070 + 2 * T4 GPUs as an example, the frame rate of the raw images captured by the 10 cameras is 15 frames per second, and the frame rate of the image processing routines inside the CPU and GPU is 113Hz.

[0145] like Figure 5A As shown, in the 500A, the frame rate of the camera acquiring the original image in the first image processing link is stable at 15Hz without any breaks. Based on the first integrity information, it is determined that there is no frame loss in the first image processing link.

[0146] like Figure 5B As shown, in 500B, the first processing frame rate in the second image processing link is stable at 113Hz without any breaks. Based on the second integrity information, it is determined that there are no frame drops in the second image processing link.

[0147] like Figure 5C As shown, in the 500C, the second processing frame rate in the third image processing link fluctuates between 80 Hz and 120 Hz without any breaks. Based on the third integrity information, it is determined that there are frame drops in the third image processing link.

[0148] Therefore, based on the abnormal link detection method, the third image processing link is determined to be an abnormal image processing link, and the abnormal problem is at least one of the system software and driver processing abnormalities.

[0149] According to embodiments of this disclosure, the above-described abnormal link detection method may further include the following operation to obtain second transmission link status information.

[0150] Determine at least one of the third transmission link status information and bandwidth margin information. Obtain the second transmission link status information based on at least one of the third transmission link status information and bandwidth margin information.

[0151] According to embodiments of this disclosure, third transmission link status information can characterize the transmission link status of the PCIe transmission link. Bandwidth margin information can characterize the bandwidth margin of the PCIe.

[0152] According to embodiments of this disclosure, the second image processing link can transmit the second intermediate image to the third image processing link via a PCIe transmission link. The second transmission link status information may include third transmission link status information characterizing whether the PCIe transmission link is functioning correctly, and bandwidth margin information of the PCIe transmission link.

[0153] According to embodiments of this disclosure, the third transmission link status information may include whether an Advanced Error Reporting (AER) error exists on the PCIe transmission link, i.e., an AER error. Bandwidth margin information may include total bandwidth and bandwidth utilization. Furthermore, bandwidth margin information may also include bandwidth utilization rate, etc.

[0154] According to embodiments of this disclosure, obtaining second transmission link status information based on at least one of third transmission link status information and bandwidth margin information may include: obtaining the second transmission link status information as normal if the PCIe transmission link is determined to be normal based on the third transmission link status information and the bandwidth margin information is determined to be sufficient based on the bandwidth margin information; and obtaining the second transmission link status information as abnormal if the PCIe transmission link is determined to be abnormal based on the third transmission link status information and the bandwidth margin information is determined to be insufficient based on the bandwidth margin information.

[0155] For example, if the third transmission link status information indicates an AER error, and the bandwidth margin information is 32Gbps total bandwidth and 33Gbps bandwidth utilization, this suggests insufficient bandwidth margin and an anomaly in the PCIe transmission link, thus indicating an anomaly in the second transmission link status information. Based on the second transmission link status information, it is determined that the third image processing link is an abnormal transmission link.

[0156] According to embodiments of this disclosure, by processing the current detection information of the current image processing link, including integrity information, transmission link status information and cache status information, it is determined whether the current image processing link is a normal image processing link, and then abnormal image processing links are identified from the entire image processing link. Since the abnormal location narrows the scope of the problem, the abnormal problem that generates abnormal images can be quickly resolved. This not only improves the reliability and accuracy of abnormal link location, but also improves the detection efficiency and resource utilization of abnormal links.

[0157] The above are merely exemplary embodiments, but are not limited thereto. Other abnormal link detection methods known in the art may also be included, as long as they can improve the detection efficiency of abnormal image processing links.

[0158] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0159] Figure 6 A block diagram of an abnormal link detection apparatus according to an embodiment of the present disclosure is shown schematically.

[0160] like Figure 6As shown, the abnormal link detection device 600 may include an acquisition module 610 and an abnormal detection module 620.

[0161] The acquisition module 610 is used to acquire autonomous driving images. Autonomous driving images are images generated during the autonomous driving process.

[0162] The anomaly detection module 620 is used to determine the abnormal image processing link from the entire image processing chain when an abnormal image is detected, based on detection information from at least one image processing link included in the entire image processing chain. The image processing links are used to process images generated during autonomous driving, and the abnormal image processing link represents the image processing link that generated the abnormal image.

[0163] According to embodiments of this disclosure, the anomaly detection module 620 includes repeatedly executing the following unit until an abnormal image processing link is determined from the entire image processing link. The anomaly detection module 620 may include a first determination submodule and a second determination submodule.

[0164] The first determining submodule is used to determine the current detection information of the current image processing link. The current image processing link represents the current level of the image processing link in the entire image processing link.

[0165] The second determining submodule is used to determine the next image processing link as the new current image processing link when the current image processing link is determined to be a normal image processing link based on the current detection information. The next image processing link is the image processing link at the next level below the current image processing link.

[0166] According to embodiments of this disclosure, the entire image processing chain includes a first image processing chain. The first image processing chain characterizes the visual sensor image processing chain.

[0167] When the current image processing link is the first image processing link, the current detection information includes at least one of the following: first integrity information, first transmission link status information, and first buffer status information. The first integrity information characterizes the integrity of the original image input to the vision sensor. The first transmission link status information characterizes the transmission link status of the vision sensor.

[0168] According to embodiments of this disclosure, the second determining submodule may further include a first determining unit.

[0169] The first determining unit is configured to determine that the first image processing link is a normal image processing link when at least one of the following is determined: the first image processing link does not have frame loss based on the first integrity information; the first image processing link is a normal transmission link based on the first transmission link status information; and the first buffer status information determines that the buffer is not full.

[0170] According to embodiments of this disclosure, the second determining submodule may further include a signal eye diagram determining unit and a first state determining unit.

[0171] The signal eye diagram determination unit is used to determine the signal eye diagram of the transmission link from the vision sensor to the central processing unit.

[0172] The first state determination unit is used to obtain the first transmission link state information based on the signal eye diagram.

[0173] According to embodiments of this disclosure, the image processing chain further includes a second image processing chain. The second image processing chain represents the central processing unit (CPU) image processing chain. The second image processing chain is the next level image processing chain after the first image processing chain.

[0174] According to embodiments of this disclosure, when the current image processing link is a second image processing link, the current detection information includes at least one of the following: second integrity information and second cache state information. The second integrity information characterizes the integrity of the first intermediate image input to the central processing unit. The first intermediate image is obtained from the original image.

[0175] According to embodiments of this disclosure, the second determining submodule may further include a second determining unit.

[0176] The second determining unit is configured to determine that the second image processing link is a normal image processing link if at least one of the following is determined: the second image processing link does not have frame loss based on the second integrity information and the second buffer status information determines that the buffer is not full.

[0177] According to embodiments of this disclosure, the second determining submodule may further include a first frame rate processing unit.

[0178] The first frame rate processing unit is used to obtain the second integrity information based on the first processing frame rate. The first processing frame rate represents the processing frame rate of the image processing routine in the central processing unit.

[0179] According to embodiments of this disclosure, the image processing chain further includes a third image processing chain. The third image processing chain represents the graphics processor image processing chain. The third image processing chain is the next level down from the second image processing chain.

[0180] According to embodiments of this disclosure, when the current image processing link is a third image processing link, the current detection information includes at least one of the following: third integrity information, second transmission link status information, and third buffer status information. The third integrity information characterizes the integrity of the second intermediate image input to the graphics processor. The second intermediate image is obtained based on the first intermediate image. The second transmission link status information characterizes the transmission link status of the graphics processor.

[0181] The second determining submodule may also include a third determining unit.

[0182] The third determining unit is used to determine that the third image processing link is a normal image processing link when at least one of the following is determined: the third image processing link does not have frame loss based on the third integrity information; the third image processing link is a normal transmission link based on the second transmission link status information; or the third buffer status information determines that the buffer is not full.

[0183] According to embodiments of this disclosure, the second determining submodule may further include a link status information and bandwidth margin determining unit and a second status determining unit.

[0184] The link status information and bandwidth margin determination unit is used to determine at least one of the third transmission link status information and bandwidth margin information. The third transmission link status information represents the transmission link status of the PCIe transmission link. The bandwidth margin information represents the bandwidth margin of the PCIe.

[0185] The second state determination unit is used to obtain the second transmission link state information based on at least one of the third transmission link state information and bandwidth margin information.

[0186] According to embodiments of this disclosure, the second determining submodule may further include a second frame rate processing unit.

[0187] The second frame rate processing unit is used to obtain the third integrity information based on the second processing frame rate. The second processing frame rate represents the processing frame rate of the image processing routine in the graphics processor.

[0188] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0189] According to an embodiment of the present disclosure, an electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method described above.

[0190] According to embodiments of the present disclosure, a non-transitory computer-readable storage medium stores computer instructions, wherein the computer instructions are used to cause a computer to perform the method described above.

[0191] According to an embodiment of this disclosure, a computer program product includes a computer program that, when executed by a processor, implements the method described above.

[0192] Figure 7The diagram schematically illustrates an electronic device suitable for implementing an abnormal link detection method according to embodiments of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0193] like Figure 7 As shown, device 700 includes a computing unit 701, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 702 or a computer program loaded into random access memory (RAM) 703 from storage unit 708. The RAM 703 may also store various programs and data required for the operation of device 700. The computing unit 701, ROM 702, and RAM 703 are interconnected via bus 704. Input / output (I / O) interface 705 is also connected to bus 704.

[0194] Multiple components in device 700 are connected to I / O interface 705, including: input unit 706, such as keyboard, mouse, etc.; output unit 707, such as various types of monitors, speakers, etc.; storage unit 708, such as disk, optical disk, etc.; and communication unit 709, such as network card, modem, wireless transceiver, etc. Communication unit 709 allows device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0195] The computing unit 701 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as the abnormal link detection method. For example, in some embodiments, the abnormal link detection method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the abnormal link detection method described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform the abnormal link detection method by any other suitable means (e.g., by means of firmware).

[0196] Based on the aforementioned electronic devices, this disclosure also provides an autonomous driving vehicle, which may include electronic devices, and may further include communication components, a display screen for implementing a human-machine interface, and information collection devices for collecting information about the surrounding environment, etc., wherein the communication components, display screen, information collection devices, and electronic devices are communicatively connected. The electronic devices included in the autonomous driving vehicle can implement the object motion trajectory information processing method described in the embodiments of this disclosure.

[0197] Among them, electronic devices can be integrated with communication components, displays, and information acquisition devices, or they can be set up separately from communication components, displays, and information acquisition devices.

[0198] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0199] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0200] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0201] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0202] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0203] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, distributed system servers, or servers incorporating blockchain technology.

[0204] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0205] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. An abnormal link detection method, comprising: Acquire autonomous driving images, wherein the autonomous driving images are images generated during the autonomous driving process; and If the autonomous driving image is detected to be an abnormal image, an abnormal image processing link is determined from the image processing chain based on the detection information of at least one image processing link included in the image processing chain. The image processing link is used to process the image generated during the autonomous driving process, and the abnormal image processing link represents the image processing link that generated the abnormal image. The step of determining an abnormal image processing link from the entire image processing chain based on detection information of at least one image processing link included in the entire image processing chain includes repeatedly performing the following operations until the abnormal image processing link is determined from the entire image processing chain: Determine the current detection information of the current image processing link, wherein the current detection information includes integrity information, the integrity information representing the integrity of the image input to the current image processing link; and If, based on the current detection information, it is determined that the current image processing link is a normal image processing link, the next image processing link is determined as the new current image processing link. When the current image processing link is the first image processing link, the current detection information includes first integrity information, which is determined based on the continuity and stability of the frame rate in the frame rate images acquired by the visual sensor. When the current image processing link is the second image processing link, the current detection information includes second integrity information, which is obtained based on the first processing frame rate, and the first processing frame rate represents the processing frame rate of the image processing routine in the central processing unit. When the current image processing link is the third image processing link, the current detection information includes third integrity information, which is obtained based on the second processing frame rate, which represents the processing frame rate of the image processing routine in the image processor.

2. The method according to claim 1, wherein, The image processing end link includes a first image processing link, which represents the visual sensor image processing link. When the current image processing link is the first image processing link, the current detection information further includes at least one of the following: first transmission link status information and first buffer status information, wherein the first transmission link status information characterizes the transmission link status of the visual sensor; The method further includes: If at least one of the following conditions is met—that the transmission link of the first image processing link is a normal transmission link based on the first transmission link status information and that the cache is not full based on the first cache status information—the first image processing link is determined to be the normal image processing link.

3. The method according to claim 2, further comprising: Determine the signal eye diagram of the transmission link from the vision sensor to the central processing unit; as well as The first transmission link status information is obtained based on the signal eye diagram.

4. The method according to claim 2 or 3, wherein, The image processing link also includes a second image processing link, which represents the central processing unit image processing link and is the next level image processing link after the first image processing link. When the current image processing link is the second image processing link, the current detection information further includes: second cache status information; The method further includes: If the cache is determined to be not full based on the second cache status information, the second image processing link is determined to be the normal image processing link.

5. The method according to claim 4, wherein, The image processing chain also includes a third image processing chain, which represents the graphics processor image processing chain and is the next level image processing chain after the second image processing chain. When the current image processing link is the third image processing link, the current detection information further includes at least one of the following: second transmission link status information and third cache status information, wherein the second transmission link status information characterizes the transmission link status of the graphics processor; The method further includes: If at least one of the following conditions is met—that the transmission link of the third image processing link is a normal transmission link based on the second transmission link status information and that the cache is not full based on the third cache status information—the third image processing link is determined to be the normal image processing link.

6. The method according to claim 5, further comprising: Determine at least one of third transmission link status information and bandwidth margin information, wherein the third transmission link status information characterizes the transmission link status of the high-speed serial computer expansion bus standard PCIe transmission link, and the bandwidth margin information characterizes the bandwidth margin of the PCIe; and The second transmission link status information is obtained based on at least one of the third transmission link status information and the bandwidth margin information.

7. An abnormal link detection device, comprising: An acquisition module is used to acquire autonomous driving images, wherein the autonomous driving images are images generated during the autonomous driving process; and An anomaly detection module, when detecting that the autonomous driving image is an anomalous image, determines an anomalous image processing link from the entire image processing chain based on detection information of at least one image processing link included in the entire image processing chain, wherein the image processing link is used to process the image generated during the autonomous driving process, and the anomalous image processing link represents the image processing link that generated the anomalous image. Specifically, the step of determining an abnormal image processing link from the entire image processing chain based on detection information from at least one image processing link included in the entire image processing chain, wherein the anomaly detection module includes repeatedly executing the following unit until the abnormal image processing link is determined from the entire image processing chain: The first determining submodule is used to determine the current detection information of the current image processing link, wherein the current image processing link represents the current level of the image processing link in the entire image processing link, and the current detection information includes integrity information, which represents the integrity of the image input to the current image processing link; and The second determining submodule is used to determine the next image processing link as the new current image processing link when the current image processing link is determined to be a normal image processing link based on the current detection information. When the current image processing link is the first image processing link, the current detection information includes first integrity information, which is determined based on the continuity and stability of the frame rate in the frame rate images acquired by the visual sensor. When the current image processing link is the second image processing link, the current detection information includes second integrity information, which is obtained based on the first processing frame rate, and the first processing frame rate represents the processing frame rate of the image processing routine in the central processing unit. When the current image processing link is the third image processing link, the current detection information includes third integrity information, which is obtained based on the second processing frame rate, which represents the processing frame rate of the image processing routine in the image processor.

8. The apparatus according to claim 7, wherein, The image processing end link includes a first image processing link, which represents the visual sensor image processing link. When the current image processing link is the first image processing link, the current detection information further includes at least one of the following: first transmission link status information and first buffer status information, wherein the first transmission link status information characterizes the transmission link status of the visual sensor; The second determining submodule also includes: The first determining unit is configured to determine that the first image processing link is the normal image processing link when at least one of the following is determined: the transmission link of the first image processing link is a normal transmission link based on the first transmission link status information and the cache is not full based on the first cache status information.

9. The apparatus according to claim 8, further comprising: A signal eye diagram determination unit is used to determine the signal eye diagram of the transmission link from the vision sensor to the central processing unit; as well as The first state determination unit is used to obtain the first transmission link state information based on the signal eye diagram.

10. The apparatus according to claim 8 or 9, wherein, The image processing link also includes a second image processing link, which represents the central processing unit image processing link and is the next level image processing link after the first image processing link. When the current image processing link is the second image processing link, the current detection information further includes: second cache status information; The second determining submodule also includes: The second determining unit is used to determine that the second image processing link is the normal image processing link when the cache is determined to be not full based on the second cache status information.

11. The apparatus according to claim 9 or 10, wherein, The image processing chain also includes a third image processing chain, which represents the graphics processor image processing chain and is the next level image processing chain after the second image processing chain. When the current image processing link is the third image processing link, the current detection information further includes at least one of the following: second transmission link status information and third cache status information, wherein the second transmission link status information characterizes the transmission link status of the graphics processor; The second determining submodule also includes: The third determining unit is configured to determine that the third image processing link is the normal image processing link when at least one of the following is determined: the transmission link of the third image processing link is a normal transmission link based on the second transmission link status information and the cache is not full based on the third cache status information.

12. The apparatus of claim 11, further comprising: A link status information and bandwidth margin determination unit is used to determine at least one of third transmission link status information and bandwidth margin information, wherein the third transmission link status information characterizes the transmission link status of the high-speed serial computer expansion bus standard PCIe transmission link, and the bandwidth margin information characterizes the bandwidth margin of the PCIe; and The second state determination unit is used to obtain the second transmission link state information based on at least one of the third transmission link state information and the bandwidth margin information.

13. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 6.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1 to 6.

15. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 6.

16. An autonomous vehicle, including the electronic equipment as claimed in claim 13.

Images