Image processing method, image processing apparatus, electronic device, computer-readable storage medium, and computer program

Abstract

The embodiments of the present disclosure disclose an image processing method and its device, electronic device, storage medium, and computer program product, which includes: acquiring a video stream collected by an image collecting device installed on a traveling device, determining a plurality of frame images including a specific traffic object from the video stream, determining a category of the specific traffic object in each frame image of the plurality of frame images and a reliability of the category, and determining correction information of a category whose reliability does not meet a preset condition based on a comparison result of the reliability of the category of the specific traffic object in the plurality of frame images.

Description

Technical Field

[0001] (Cross - reference to related applications) This disclosure is based on and claims priority to a Chinese patent application with application number 202210301591.0, filing date of March 24, 2022, and application title "Image Processing Method and Its Apparatus, Electronic Device, and Storage Medium", and all contents of the Chinese patent application are incorporated herein by reference.

[0002] This disclosure relates to the field of image processing technology, but is not limited thereto. Specifically, it relates to an image processing method and its apparatus, an electronic device, a storage medium, and a computer program product.

Background Art

[0003] In related technologies, the classification and identification of traffic objects (such as traffic signs) can be realized by a multi - classification task model. However, there are many traffic objects and they are easily affected by factors such as long distances and object occlusion, resulting in the problem of low identification accuracy.

Summary of the Invention

Means for Solving the Problems

[0004] In view of this, embodiments of the present disclosure provide an image processing method and its apparatus, an electronic device, a storage medium, and a computer program product. [[ID=зо]]

[0005] The technical solutions of the embodiments of the present disclosure are realized as follows.

[0006] Embodiments of the present disclosure provide an image processing method, and the image processing method includes: Obtaining a video stream collected by an image collection device installed on a traveling device, and determining a plurality of frame images including a specific traffic object from the video stream; Determining the category of a specific traffic object in each frame image of the plurality of frame images and the confidence level of the category; This includes determining correction information for categories whose reliability does not meet pre-set conditions, based on the results of comparing the reliability of specific traffic object categories within the aforementioned multiple frame images.

[0007] Embodiments of the present disclosure further provide an image processing apparatus, the image processing apparatus comprising an acquisition unit, a first decision unit, a second decision unit, and a third decision unit, The acquisition unit is configured to acquire a video stream collected by an image acquisition device installed on the vehicle. The first decision unit is configured to determine a plurality of frame images containing a specific traffic object from the video stream, The second decision unit is configured to determine the confidence level of a specific traffic object category in each frame of the plurality of frame images, The third decision unit is configured to determine correction information for categories whose reliability does not meet pre-set conditions, based on the results of comparing the reliability of specific traffic object categories within the plurality of frame images.

[0008] When the embodiments of the present disclosure are executed by a processor, the present invention further provides a computer-readable storage medium storing a computer program for the processor to implement the steps of the image processing method described in the embodiments of the present invention.

[0009] Embodiments of the present disclosure further provide an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the steps of the image processing method described in the embodiment of the present disclosure.

[0010] Embodiments of the present disclosure provide a computer program product comprising a non-temporary computer-readable storage medium on which the computer program is stored, and which, when the computer program is read and executed by a computer, implements some or all of the steps in the image processing method described in the embodiments of the present disclosure.

[0011] Embodiments of this disclosure provide an image processing method and apparatus, electronic equipment, storage medium, and computer program product. The method includes acquiring a video stream collected by an image acquisition device installed on a vehicle, determining a plurality of frame images containing a specific traffic object from the video stream, determining the category of a specific traffic object in each frame of the plurality of frame images and the confidence level of the category, and determining correction information for categories whose confidence level does not meet a predetermined condition based on a comparison of the confidence levels of the categories of specific traffic objects in the plurality of frame images. By adopting the technical solution of the embodiment of this disclosure, categories whose confidence level does not meet a predetermined condition based on a comparison of the confidence levels of the categories of specific traffic objects in the plurality of frame images can be corrected, that is, by correcting low-reliability classification results with high-reliability classification results for a specific traffic object, the accuracy of classification of traffic objects in images can be improved on the one hand, and a basis for confidence in downstream policy decision control can be provided on the other hand, and subsequent real-time control can be facilitated. [Brief explanation of the drawing]

[0012] [Figure 1] This is a schematic diagram illustrating an application scenario of the image processing method according to the embodiments of this disclosure. [Figure 2] This is a schematic flowchart of the image processing method according to the embodiments of this disclosure. [Figure 3] This is a schematic diagram of the classification result in the image processing method according to the embodiments of the present disclosure. [Figure 4] This is a schematic diagram illustrating the reliability of the classification results in the image processing method according to the embodiments of this disclosure. [Figure 5] This is a schematic diagram of the structure of an image processing apparatus according to an embodiment of the present disclosure. [Figure 6] This is a schematic structural diagram of the hardware configuration of an electronic device according to an embodiment of the present disclosure. [Modes for carrying out the invention]

[0013] Embodiments of this disclosure will be described in more detail below with reference to the drawings and specific examples.

[0014] In related technologies, the identification of traffic objects is mainly achieved by a single-layer multi-classifier, and there are problems such as the difficulty in labeling and the inability to achieve accurate classification due to the large number of categories of traffic objects, the influence of long distance and object occlusion. Figure 1 is a schematic diagram of the application scenario of the image processing method according to an embodiment of this disclosure. As shown in Figure 1, in the image of frame 21, the distance is too far and the contents of the traffic sign cannot be identified. In the image of frame 51, the traffic sign is easily identified as a speed limit of 30 because it is obscured by the tree trunk. Similarly, in the image of frame 55, the traffic sign is easily identified as a speed limit of 60 because it is obscured by the tree trunk. Finally, in the image of frame 60, the traffic sign can be correctly identified as a speed limit of 50.

[0015] In embodiments of this disclosure, the electronic device can improve the accuracy of classifying traffic objects in an image by correcting categories whose reliability does not meet pre-set conditions based on a comparison of the reliability of categories of specific traffic objects in multiple frame images, that is, by correcting low-reliability classification results of specific traffic objects with high-reliability classification results of specific traffic objects, thereby providing a basis for confidence in downstream policy decision control.

[0016] In each embodiment of the present disclosure, the traffic object may be any object on the road, and may include, for example, at least one of the following objects: a traffic sign, a road sign, a traffic participant, and a traffic signal light.

[0017] In embodiments of the present disclosure, it should be noted that terms such as "comprising" and "including" or any other variations thereof are intended to cover non-exclusive inclusion, such that a method or apparatus comprising a series of elements includes not only the explicitly recited elements but also other elements not explicitly listed or elements inherent to the implementation method or apparatus. Without further limitation, an element limited by the phrase "comprising one..." does not exclude the further presence of other related elements in the method or apparatus comprising the element (for example, steps in the method or units in the apparatus, where the unit can be, for example, a part of a circuit, a part of a processor, a part of a program or software, etc.).

[0018] For example, the image processing method provided by embodiments of the present disclosure includes a series of steps, but the image processing method provided by embodiments of the present disclosure is not limited to the described steps. Similarly, the image processing apparatus provided by embodiments of the present disclosure includes a series of modules, but the apparatus provided by embodiments of the present disclosure is not limited to including the explicitly recorded modules and may further include modules that need to be set when obtaining related information or performing processing based on the information.

[0019] In this specification, the term "and / or" is only used to explain the relationship of related objects and indicates that three types of relationships can exist. For example, A and / or B can indicate three situations: A exists alone, A and B exist simultaneously, and B exists alone. Also, in this specification, the term "at least one" indicates any one of a plurality or any combination of at least two of a plurality. For example, including at least one of A, B, and C can indicate including any one or more elements selected from the set consisting of A, B, and C.

[0020] Embodiments of the present disclosure provide an image processing method. FIG. 2 is a schematic flowchart of the image processing method according to an embodiment of the present disclosure. As shown in FIG. 2, the method includes steps 101 to 103.

[0021] In step 101, obtain a video stream collected by an image collection device installed in a traveling device, and determine a plurality of frame images including specific traffic targets from the video stream.

[0022] In step 102, determine the category of a specific traffic target in each frame image of the plurality of frame images and the confidence level of the category.

[0023] In step 103, based on the comparison result of the confidence levels of the categories of specific traffic targets in the plurality of frame images, determine correction information for categories that do not meet the preset conditions of the confidence level.

[0024] The image processing method of this embodiment is applied to an electronic device, and the electronic device may be an in-vehicle device, or may be a cloud platform or other computer devices. Exemplarily, the in-vehicle device may be a sink client, a thick client, a microprocessor-based system, a small computer system, etc. installed in a traveling device, and the cloud platform may be a distributed cloud computing technology environment including a small computer system or a large computer system. Here, the traveling device may be, for example, various vehicles traveling on a road. In the following embodiments, it is exemplified that the traveling device is a vehicle in all cases.

[0025] In this embodiment, the in-vehicle equipment can communicate with vehicle sensors, positioning devices, etc., and through this communication connection, the in-vehicle equipment can acquire data collected by the vehicle sensors and geographic location information reported by the positioning devices. For example, the vehicle sensors may be at least one of the following: millimeter-wave radar, laser radar, camera, etc. The positioning device may be a device for providing positioning services based on at least one of the following: Global Positioning System (GPS), Beidou satellite navigation system, or Galileo satellite navigation system.

[0026] For example, the in-vehicle equipment may be an Advanced Driving Assistant System (ADAS), which is installed in the vehicle and may acquire real-time vehicle location information from the vehicle's positioning device, and / or may acquire image data, radar data, etc., representing information about the vehicle's surrounding environment from the vehicle's sensors. Here, optionally, the ADAS can transmit vehicle driving data, including the vehicle's real-time location information, to a cloud platform, and in this way, the cloud platform can receive the vehicle's real-time location information and / or image data, radar data, etc., representing information about the vehicle's surrounding environment.

[0027] In this embodiment, the video stream is acquired by an image acquisition device (i.e., the aforementioned sensor such as a camera) set on the vehicle, and the image acquisition device collects road images or environmental images around the vehicle in real time as the vehicle moves. That is, the video stream may be a series of images obtained by continuously collecting the surrounding environment or scenario as the vehicle moves.

[0028] In some selectable embodiments, the electronic device can use a classification network to identify the image of each frame in the video stream, determine a specific traffic object contained in the image of each frame, and determine the category of that traffic object. Exemplarily, the video stream can be used as input data for the classification network, the classification network can perform feature extraction on the image of each frame in the video stream, determine a specific traffic object in the image based on the extracted features, determine a first region of the specific traffic object in the image, and determine the category of the specific traffic object. Here, the category of the specific traffic object may be one of several classifications of traffic objects. For example, each traffic object may be pre-divided into several categories, and each category may contain one or more traffic objects. The category to which a specific traffic object belongs may be one of the aforementioned pre-divided categories.

[0029] In some selectable embodiments, determining the confidence level of a particular traffic object category in each frame of the plurality of frame images may include determining a template image for the particular traffic object category in each frame of the plurality of frame images, calculating the similarity between the particular traffic object and the template image, and determining the confidence level of the particular traffic object category based on the similarity.

[0030] In this embodiment, template images for each category are stored in the electronic device. After determining the category of a specific traffic object, the electronic device compares an image of the first region where the specific traffic object is located with the template image of the corresponding category, calculates the similarity between the specific traffic object and the template image of the corresponding category, and uses the calculated similarity as the confidence level for the category of the specific traffic object.

[0031] In this embodiment, the electronic device determines correction information for categories whose reliability does not meet a predetermined condition, based on the comparison results of the reliability of specific traffic object categories within the plurality of frame images. That is, it corrects categories whose reliability does not meet a predetermined condition. Here, exemplarily, failing to meet a predetermined condition may mean that the reliability is less than a predetermined threshold.

[0032] Optionally, determining correction information for categories whose reliability does not meet a predetermined condition based on a comparison of the reliability of categories of specific traffic objects in the plurality of frame images includes setting the category of the specific traffic object in the second image to the category of the specific traffic object in the first image in response to the reliability of a category of a specific traffic object in the first image of the plurality of frame images meeting a predetermined condition, and the reliability of a category of the specific traffic object in the second image of the plurality of frame images not meeting the predetermined condition. Here, the order of the first and second images is not limited, i.e., the first image may be after the second image, or the first image may be before the second image.

[0033] In this embodiment, the electronic device can use the highly reliable classification result for a specific traffic object to correct the low-reliability classification result for the same traffic object, thereby correcting categories whose reliability does not meet the pre-set conditions, providing sufficient justification to downstream modules (e.g., control modules, policy decision modules, etc.) and facilitating subsequent real-time control.

[0034] In some selectable embodiments of the present disclosure, determining a plurality of frame images containing a particular traffic object from the video stream includes determining a first region in the image of the traffic object within the image containing the traffic object in the video stream, determining a second region within the first region such that the second region is smaller than the first region, and selecting, based on the information of each second region, images containing traffic objects of a first category from the image containing the traffic object as a plurality of frame images containing a particular traffic object, wherein the first category is the category to which the particular traffic object belongs.

[0035] In this embodiment, the electronic device determines a first region in the image of the object being traveled within the image of each frame in the video stream, that is, it obtains a detection frame (e.g., a rectangular frame) in the image of the object being traveled, and the region in the image of the detection frame is the first region. Furthermore, it determines a second region within the first region in each image included in the video stream.

[0036] Optionally, the second region is the central region of the first region, and the information in the second region is the information in the central region of the feature map of the first region.

[0037] For example, the method for determining the second region may involve proportionally reducing the length and width of the first region (i.e., the detection frame for the traffic object), and making the reduced region the second region, which may also be called the central region of the first region. Another example is that the method for determining the second region may involve disproportionately reducing the length and width of the first region according to the degree to which the traffic object is obstructed, and then moving the center point according to the obstructed position of the traffic object after reduction, thereby obtaining the second region. Taking the image of frame 51 or frame 55 in Figure 1 as an example, by detecting the traffic object, it is discovered that the left side of the traffic object is obstructed, so the length and width of the first region (i.e., the detection frame for the traffic object) where the traffic object is located are disproportionately reduced, and after reduction, since the left side of the traffic object is obstructed, the reduced region is moved to the right (the moved region is also within the first region), thereby obtaining the second region. In this way, as many features of the traffic object as possible are left in the second region, and the features of the obstructing object are reduced.

[0038] In this embodiment, the electronic device screens images containing traffic objects of a first category from images containing traffic objects based on information in a second region of the image, and combines multiple frames containing traffic objects of the first category into multiple frame images containing a specific traffic object, wherein the first category is the category to which the specific traffic object belongs.

[0039] In some selectable embodiments, selecting an image containing a traffic object of a first category from an image containing the traffic object, based on information from each of the second regions, includes: performing feature extraction on each of the second regions, determining a first similarity of the pixel points of each second region based on the extracted features, and determining an image containing a traffic object of the first category where the location information satisfies a first preset condition and the first similarity satisfies a second preset condition. Here, the first category is the category to which a particular traffic object belongs.

[0040] Here, optionally, the positional information of a second region in an image containing the traffic object satisfies a first preset condition if the distance between the positional information of the second region in any two adjacent frames of the image containing the traffic object, i.e., the difference between the positions of the second region in the corresponding images, is less than a first threshold. Here, the distance may be the distance in a specified coordinate system (e.g., pixel coordinate system, image coordinate system, etc.).

[0041] Optionally, the statement that the first similarity satisfies the second pre-set condition can mean that the first similarity is equal to or greater than the second threshold.

[0042] As an example, suppose the image containing the traffic object includes a first image and a second image, where the second image is an image from a frame after the first image. In one example, the second image may be an image from a frame after the first image. In another example, the second image may be an image from several frames after the first image. For example, in Figure 1, the first image may be the image from the 21st frame, and the second image may be the image from the 51st, 55th, or 60th frame.

[0043] The electronic device performs traffic object identification on the first and second images, respectively, and determines the first region of the traffic object in the first image and the first region of the traffic object in the second image. In this example, tracking the identified traffic object in the first image involves tracking using a second region that is smaller than the first region where the traffic object is located. Considering the occlusion situation, the edges of the traffic object are usually obscured, so tracking using the second region is robust to occlusion.

[0044] For example, the length and width of the first region (i.e., the detection frame) may be reduced proportionally, and the reduced region may be called the second region, which may be referred to as the central region of the first region. In some embodiments, it is possible to determine whether the traffic object in the first image and the second image is a traffic object of the first category based on the pixel points of the second region in the first image and the pixel points of the second region in the second image.

[0045] In some embodiments, feature extraction processing can be performed on the pixel points of the second region in the first image, and feature extraction processing can be performed on the pixel points of the second region in the second image, and a similarity between the two (referred to as the first similarity) can be calculated based on the extracted features. If the first position information and the second position information satisfy a first preset condition, and the first similarity satisfies a second preset condition, it is determined that the traffic object corresponding to the second region in the first image and the second image belongs to the first category. Here, the first position information may be the coordinates of the center point of the second region in the first image, and the second position information may be the coordinates of the center point of the second region in the second image.

[0046] Here, optionally, the first and second position information satisfying a first preset condition may be that the distance between the first and second position information (for example, the distance between the center point coordinates of the second region in the first image and the center point coordinates of the second region in the second image) is less than a first threshold. Here, the distance may be the distance in a specified coordinate system (for example, a pixel coordinate system, an image coordinate system, etc.). For example, in the pixel coordinate system, the first center point coordinates corresponding to the first position information are determined, the second center point coordinates corresponding to the second position information are determined, and the distance between the first and second position information is obtained by subtracting the first and second center point coordinates. In actual application, because the frame interval of image acquisition is extremely small, if traffic objects of the first category are included in different frame images, the positions of the traffic objects in different frame images are also close.

[0047] Optionally, the condition that the first similarity satisfies the second pre-set condition may also be that the first similarity is equal to or greater than the second threshold. If the first similarity is equal to or greater than the second threshold, it can be determined that the traffic object in the second region of the first image and the traffic object in the second region of the second image belong to the same category, i.e., the first category (but they may not be the same traffic object). Next, by combining the fact that the first and second location information satisfies the first pre-set condition, it can be determined that the traffic object corresponding to the second region of the first image and the traffic object corresponding to the second region of the second image are the same traffic object.

[0048] In one case, feature extraction can be performed on the second region within each image, and a first similarity can be determined for the pixel points of the second region within each image based on the extracted features. If the first similarity satisfies a second pre-set condition, it can be determined that the traffic objects in each image belong to the same category (e.g., the first category). In this case, it can be determined that the traffic objects in each image belong to the same category (e.g., the first category), but may not be the same traffic object. In another case, based on determining the first similarity described above, if the positional information of the second region within each image satisfies a first pre-set condition, and the first similarity satisfies a second pre-set condition, it can be determined that the traffic objects in each image belong to the same category (e.g., the first category). In this case, it can be determined that the traffic objects in each image belong to the same category (e.g., the first category) and are also the same traffic object. In other words, the traffic objects of the first category described in this embodiment are not limited to traffic objects in at least two frames belonging to the same category, but can also include situations where traffic objects in at least two frames are the same traffic object.

[0049] During implementation, it can be determined that the traffic object in at least two frames of images is the same traffic object by the following solution: For each traffic object identified in the image of each frame, a single corresponding identifier (ID) can be assigned to facilitate the identification of the same traffic object across different frame images, and in response to situations where traffic objects in multiple frame images belong to the same category (e.g., Category 1), the Category 1 identifier assigned to the traffic object in the multiple frame images is associated.

[0050] In this embodiment, taking as an example that multiple frame images include a first image and a second image, after identifying a traffic object (e.g., denoted as object 1) in the first image, a first identifier is assigned to the traffic object (object 1). If it is determined that a traffic object (e.g., denoted as object 2) corresponding to the second region in the second image and a traffic object (object 1) corresponding to the second region in the first image belong to the same category (e.g., category 1), then the first identifier assigned to object 1 may also be associated with object 2. That is, both object 1 and object 2 are associated with the first identifier and are therefore traffic objects of the same category (e.g., category 1). Alternatively, if the first identifier is assigned to object 1 and the second identifier is assigned to object 2, and it is determined that object 1 and object 2 belong to the same category (e.g., category 1), then the first identifier and the second identifier are associated, and for example, the second identifier may be replaced with the first identifier. That is, both object 1 and object 2 are associated with the first identifier and are therefore traffic objects of the same category (e.g., category 1).

[0051] In some select embodiments of the present disclosure, determining a category of a particular traffic object in each frame of the plurality of frame images and the confidence level of the category includes determining the subdivision category with the highest confidence level of a first category of traffic object in each frame of the plurality of frame images and the confidence level of the subdivision category, wherein the first category is the category to which the particular traffic object belongs.

[0052] Correspondingly, determining correction information for categories whose reliability does not meet the pre-set conditions based on the comparison results of the reliability of categories of specific traffic objects in the plurality of frame images includes determining correction information for the subcategories of traffic objects in the first category whose maximum reliability does not meet the pre-set conditions based on the comparison results of the reliability of the subcategory with the highest reliability of traffic objects in the first category in the plurality of frame images.

[0053] In this embodiment, the electronic device first determines a first category of traffic objects (i.e., a broad classification category), then determines a subdivided category of traffic objects within the first category, that is, first performs a coarse-grained classification on the traffic objects, and then performs a fine-grained classification on the traffic objects within the coarse-grained classification.

[0054] Optionally, the electronic device can identify the image of each frame in the video stream via the first layer network, determine that the image of each frame contains traffic objects of the same category (first category), that is, detect traffic objects in the image of each frame, and determine that the detected traffic objects belong to the same category (i.e., the first category). Exemplarily, the video stream can be used as input data for the first layer network, feature extraction can be performed on the image of each frame in the video stream via the first layer network, traffic objects in the image of each frame can be determined based on the extracted features, a first region of the traffic object in the image of each frame can be determined, the category of the traffic object (e.g., a broad classification category) can be determined, that is, the detection frame of the traffic object in the image of each frame and the category (broad classification category) to which the traffic object belongs can be output, and further multiple frame images of traffic objects belonging to the same category (here referred to as the first category) can be determined from among them. Here, optionally, the multiple frame images may be consecutive or discontinuous frame images in the video stream. For example, a video stream may contain 100 frames, and the determined multiple frame images containing the traffic object of the first category may be frames 10 through 50, or frames 5, 15, 25, 35, 45, etc., out of the 100 frames, and are not limited to these in this embodiment.

[0055] In some selectable embodiments, the category to which the particular traffic object belongs (including the first category) is one of several traffic object classifications. Understandably, the first layer network is pre-trained and acquired based on traffic object classifications, and processing of images by the first layer network can determine whether a traffic object in an image belongs to a pre-labeled traffic object classification, and to which traffic object classification it belongs.

[0056] As an example, taking traffic objects as traffic signs (including, for example, traffic sign boards and road signs), since there are many categories of traffic signs, this embodiment preclassifies various traffic signs. For example, as shown in Figure 3, traffic signs can be preclassified into multiple first categories 41 such as speed class signs, sidewalk class signs, warning class signs, and stop class signs. Assuming that traffic objects are identified in the image of each frame in the video stream, multiple frame images containing traffic objects in the "speed class sign" category can be screened. In actual applications, each traffic sign can be classified according to its function or action. Other classification methods can be used in other embodiments, but are not limited to this embodiment.

[0057] In some embodiments, after multiple frame images containing traffic objects of the same category (i.e., the first category) are determined via a first-layer network, a second-layer network is used to perform a subclassification process on the traffic objects of the first category within each frame image, thereby obtaining the subdivided categories of the traffic objects of the first category within each frame image and the confidence level of those subdivided categories.

[0058] In this embodiment, the second layer network may be a classification network corresponding to the category to which the traffic object belongs. Optionally, the number of second layer networks can correspond to the number of categories to which the traffic object belongs; that is, each category to which a traffic object belongs can correspond to one second layer network, and each second layer network has pre-labeled sub-categories within the category to which the corresponding traffic object belongs. Taking the speed class sign shown in Figure 3 as an example, the speed class sign can include multiple sub-categories 42, such as a speed limit sign of 80 kilometers per hour (km / h), a speed limit sign of 40 km / h, a speed limit sign of 120 km / h, and a speed limit sign of 70 km / h. After determining that a traffic object is a speed class sign, the classification process of the corresponding second layer network can obtain the sub-category with the highest confidence level for the traffic object and the confidence level of that sub-category. For example, the sub-category with the highest confidence level for the traffic object may be a 70 km / h speed limit sign.

[0059] In other embodiments, the second layer network may also correspond to categories to which multiple traffic objects belong. For example, if the traffic objects are traffic signs, the second layer network may be used to identify sub-categories in “One-Class Signs,” “Turn-Class Signs,” and “Lane-Class Signs.” Alternatively, the second layer network may include multiple branch networks for classification processing, each branch network may be used to identify sub-categories corresponding to categories to which one or more classes of traffic objects belong. Exemplarily, after identifying the category to which the traffic object belongs (e.g., a first category), the electronic device extracts a sub-image corresponding to the first region where the traffic object is located, and the sub-image is input into the branch network corresponding to the first category to be used to identify sub-categories within the first category.

[0060] In this way, by first determining the first category (a broad classification category) of traffic objects, and then determining the subdivided categories of traffic objects within the first category, that is, by first performing coarse-grained classification on traffic objects, and then performing fine-grained classification on traffic objects within that coarse-grained classification, the classification accuracy of traffic objects in an image (e.g., traffic signs, road signs, etc.) can be improved. In particular, when identifying traffic objects with many categories using a single-layer multi-classifier, the problem of difficulty in labeling due to the large number of categories and inability to classify accurately can be improved.

[0061] In some selectable embodiments of the present disclosure, determining the subdivision category with the highest confidence level for a traffic object of a first category in each frame of the plurality of frame images, and the confidence level of said subdivision category, includes determining a second similarity between the traffic object of a first category in each frame of the plurality of frame images and the template image of each second category, where each second category is a subdivision category of the first category, and determining, based on the second similarity, the subdivision category with the highest confidence level for a traffic object of a first category in each frame of the image, and the confidence level of said subdivision category.

[0062] In this embodiment, template images for each second category are stored in the electronic device. After determining the first category of the object being transported, the electronic device compares the image (which may be a feature map of the region where the object is located) with the template images for each second category and determines the similarity (referred to here as the second similarity) between the object of the first category in the image for each frame and the template images for each second category. Here, the maximum second similarity may be the confidence level of the subdivided category of the object being transported (e.g., the second category), or the confidence level of the subdivided category of the object being transported (e.g., the second category) may be calculated based on the maximum second similarity.

[0063] As an example, as shown in Figure 4, the traffic object in the image of frame 60 is compared with the template image of each second category, and it is determined that the second similarity with "50 km / h speed limit sign" is 100%. The traffic object in the image of frame 55 is compared with the template image of each second category, and it is determined that the second similarity with "60 km / h speed limit sign" is 50%. The traffic object in the image of frame 51 is compared with the template image of each second category, and it is determined that the second similarity with "30 km / h speed limit sign" is 40%. The traffic object in the image of frame 21 is compared with the template image of each second category, and it is determined that the second similarity with "No U-turn sign" is 80%. From each second similarity, the subdivision category with the highest confidence level and the confidence level of that subdivision category are determined. In the above example, the highest confidence level is 100%, and the corresponding subdivision category is "50 km / h speed limit sign".

[0064] For example, if the maximum confidence level corresponding to the second category to which a traffic object belongs in multiple frame images is equal to or greater than the third threshold, the confidence level (i.e., reliability) of the second category to which the traffic object belongs is determined to be high. Conversely, if the maximum confidence level corresponding to the second category to which a traffic object belongs in the multiple frame images is less than the third threshold, the confidence level (i.e., reliability) of the second category to which the traffic object belongs is determined to be low. As shown in Figure 4, the maximum confidence level corresponding to the second category to which the traffic object belongs in the image of frame 60 is 100%, and is considered to be highly reliable. In the image of frame 21, the maximum confidence level corresponding to the second category to which the traffic object belongs is 80%, in the image of frame 51, the maximum confidence level corresponding to the second category to which the traffic object belongs is 40%, and in the image of frame 55, the maximum confidence level corresponding to the second category to which the traffic object belongs is 50%, and all of these may be considered to be of low reliability.

[0065] It should be explained here that the above third threshold can be determined based on the actual situation. In one embodiment, the third threshold can be determined based on the maximum confidence level corresponding to the image in each frame. For example, in 100 images, if 80% of the images have a maximum confidence level of 100%, 15% have a maximum confidence level of 80%, and 5% have a maximum confidence level of 50%, then the image classification results are considered relatively reliable, and the value of the third threshold can be set relatively high, for example, to 90% to 95%. Correspondingly, if the overall calculation result of the maximum confidence level in the images is not high, the value of the third threshold can be set low. This embodiment is not limited to this.

[0066] In this embodiment, the electronic device determines correction information for the subdivided categories of traffic objects in the first category whose maximum confidence level does not meet a preset condition, based on the comparison result of the confidence levels of the subdivided categories with the highest confidence level among the multiple frame images. This corrects the subdivided categories of traffic objects in the first category whose maximum confidence level does not meet the preset condition, further improving the classification accuracy of traffic objects in the video stream and providing a basis for confidence in downstream policy decision control.

[0067] In some selectable embodiments of the present disclosure, determining correction information for a subcategory of a traffic object in the first category whose maximum confidence level does not meet a preset condition, based on a comparison of the confidence levels of the subcategory with the highest confidence level of the traffic object in the first category within the plurality of frame images, includes setting the subcategory with the highest confidence level of the traffic object in the first category within the second image to the same category as the subcategory with the highest confidence level of the traffic object in the first image within the first image, in response to the fact that the confidence level of the subcategory with the highest confidence level of the traffic object in the first category within the first image satisfies a third preset condition, and the confidence level of the subcategory with the highest confidence level of the traffic object in the first category within the second image does not satisfy the third preset condition.

[0068] Here, exemplarily, the confidence level satisfying the third pre-set condition may be the confidence level being equal to or greater than the fourth threshold.

[0069] In this embodiment, the order in which the first and second images are placed among the plurality of frame images is not limited; that is, the first image may be after the second image, or the first image may be before the second image.

[0070] In some selectable embodiments, the second image is an image of a frame after the first image. This embodiment is applicable to scenarios where real-time detection is performed on the image.

[0071] For example, if the confidence level satisfies a third pre-defined condition, it can indicate high confidence or high reliability. Conversely, if the confidence level does not satisfy the third pre-defined condition, it can indicate low confidence or low reliability. For traffic objects of the same category (first category), if the confidence level of the largest second category (i.e., subcategory) of the traffic object is high in the first image (satisfies the third pre-defined condition), and the confidence level of the largest second category (i.e., subcategory) of the traffic object in the second image is low (does not satisfy the third pre-defined condition), then the high-confidence subcategory result (i.e., largest second category) in the first image can be used to replace the low-confidence subcategory result (i.e., largest second category) in the second image.

[0072] The above embodiment applies to real-time detection and classification processes for images, such as when the classification results of later images are inaccurate due to the presence of occlusion. For example, Table 1 shows the classification representation before correction, and Table 2 shows the classification representation after correction. Referring to Table 1, real-time detection and classification of a 5-frame image yields four traffic objects with identifier IDs 1, 2, 3, and 4 respectively (where the same ID is used because the traffic objects in the previous two frames of images are the same traffic object), the first and second classifications for each traffic object, and the corresponding confidence levels. If tracking the traffic objects determines that the four traffic objects are the same traffic object, then the four traffic objects are associated with the same identifier ID "1", and the low-confidence classification results are replaced with the high-confidence classification results. As shown in Table 2, all low-confidence classification results are replaced with high-confidence classification results when the first classification is speed, the second classification is 50, and the confidence level is high.

[0073] [Table 1]

[0074] [Table 2]

[0075] In this embodiment, by using highly reliable subclassification results to modify the low-reliability subclassification results (the subclassification category with the highest reliability) of the same traffic target, it is possible to modify the subclassification categories of traffic targets in the first category whose maximum reliability does not meet the pre-set conditions, thereby providing sufficient justification to downstream modules (e.g., control modules, policy decision modules, etc.) and facilitating subsequent real-time control.

[0076] In some other selectable embodiments, the first image is an image of a frame after the second image. This embodiment is applied to a scenario in which the training data is misclassified.

[0077] In this embodiment, all of the above-mentioned images (including the first and second images) and classification results are used to train the network model. If classification errors due to distance exist, for example, in the scenario shown in Figure 1, the image in frame 21 is far away, so the identified subclassification result is low reliability (or confidence), and as the object is gradually approached, the corresponding collected image becomes clearer and the reliability of the subclassification result also changes. For example, the classification result in frame 60 is high reliability (or confidence), and to optimize the training data, the subclassification result of the earlier, lower-reliability second image can be replaced with the later, higher-reliability subclassification result of the first image.

[0078] For example, Table 3 shows the classification representation before correction, and Table 4 shows the classification representation after correction. Referring to Table 3, real-time detection and classification of the 5-frame image yields four traffic objects with identifier IDs 1, 2, 3, and 4 respectively (where the same ID is used because the traffic objects in the previous two frames are the same traffic object), the first and second categories for each traffic object, and the corresponding confidence level. If tracking the traffic objects determines that the four traffic objects are the same traffic object, then, as shown in Table 4, the four traffic objects are associated with the same identifier ID "1". For traffic objects of the same category (first category), the confidence level of the detailed classification results in the previous few frames is low (does not satisfy the third pre-set condition), while the confidence level of the detailed classification results for the same traffic object in the fifth frame is high (satisfies the third pre-set condition). Therefore, all the low-confidence detailed classification results corresponding to the previous few frames are replaced with high-confidence detailed classification results, where the first category is speed, the second category is 50, and the confidence level is high.

[0079] [Table 3]

[0080] [Table 4]

[0081] In some selectable embodiments of the present disclosure, determining correction information for a subcategory of a traffic object in the first category whose maximum confidence level does not meet a preset condition, based on a comparison of the confidence levels of the subcategory with the highest confidence level of the traffic object in the plurality of frame images, may further include outputting hint information indicating that the classification result of the traffic object cannot be determined in response to the fact that none of the confidence levels of the subcategory with the highest confidence level of the traffic object in the first category in the plurality of frame images meet a third preset condition.

[0082] In this embodiment, if, among multiple frame images, the confidence level of the subdivision category with the highest confidence level of the traffic object in the first category within each frame image does not satisfy the third pre-set condition, that is, if the confidence level of the subdivision category with the highest confidence level of the traffic object in the first category within each frame image is lower than the fourth threshold, that is, if the confidence level of the subdivision category with the highest confidence level of the traffic object in the first category within each frame image is relatively low, then hint information is output indicating that the subdivision category of the traffic object cannot be determined and the classification result cannot be determined.

[0083] For example, referring to the classification results in Table 3 above, real-time detection and classification of the 5-frame image yields four traffic objects with identifier IDs 1, 2, 3, and 4 respectively (where the same ID is used because the traffic objects in the previous two frames are the same traffic object), the first and second classifications for each traffic object, and the corresponding confidence levels. If tracking the traffic objects determines that the four traffic objects are the same traffic object, then, as shown in Table 5, the four traffic objects are associated with the same identifier ID "1". Assuming that the 4th frame image has now been collected, the classification results for the second classification corresponding to the same traffic object in the previous three frames are all low confidence, and the classification results for the second classification corresponding to the same traffic object in the 4th frame image are still low confidence (the third pre-set condition is not met), so hint information is output indicating that a classification result cannot be determined. In some embodiments, as the fifth frame image is collected, if it is detected that the classification result of the second category corresponding to the same traffic object in the fifth frame image is highly reliable (satisfies a third preset condition), then a highly reliable classification result, i.e., a detailed classification result for a speed of 50 km / h, can be output.

[0084] [Table 5]

[0085] Based on embodiments of the methods described above, embodiments of the present disclosure further provide an image processing apparatus. Figure 5 is a schematic structural diagram of an image processing apparatus according to an embodiment of the present disclosure. As shown in Figure 5, the apparatus includes an acquisition unit 21, a first decision unit 22, a second decision unit 23, and a third decision unit 24.

[0086] The acquisition unit 21 is configured to acquire a video stream collected by an image acquisition device installed in the vehicle.

[0087] The first decision unit 22 is configured to determine a plurality of frame images from the video stream that include a specific traffic object.

[0088] The second decision unit 23 is configured to determine the category of a specific traffic object in each frame of the plurality of frame images and the confidence level of the category.

[0089] The third decision unit 24 is configured to determine correction information for categories whose reliability does not meet pre-set conditions, based on the results of comparing the reliability of specific traffic object categories within the plurality of frame images.

[0090] In some selectable embodiments of the present disclosure, the first determination unit 22 determines a first region in the image of a traffic object within an image containing a traffic object in the video stream, determines a second region within the first region for each first region, wherein the second region is smaller than the first region, and based on the information of each second region, selects an image containing a traffic object of a first category from the image containing the traffic object, and is configured to make a multi-frame image containing a specific traffic object into a multi-frame image containing a specific traffic object, wherein the first category is the category to which the specific traffic object belongs.

[0091] In some select embodiments of the present disclosure, the second region is the central region of the first region, and the information of the second region is the information of the central region of the feature map of the first region.

[0092] In some selectable embodiments of the present disclosure, the first decision unit 22 is configured to perform feature extraction on each of the second regions, determine a first similarity of the pixel points of each second region based on the extracted features, and determine an image in which a second region is located such that the location information satisfies a first preset condition and the first similarity satisfies a second preset condition as an image containing traffic objects of a first category.

[0093] In some selectable embodiments of the present disclosure, the second decision unit 23 is configured to determine the subdivision category with the highest confidence level for a traffic object in a first category within each frame of the plurality of frame images, and the confidence level of the subdivision category, wherein the first category is the category to which the particular traffic object belongs.

[0094] In some selectable embodiments of the present disclosure, the third decision unit 24 is configured to determine correction information for the sub-categories of traffic objects in the first category whose maximum confidence level does not meet a preset condition, based on a comparison of the confidence levels of the sub-categories of traffic objects in the plurality of frame images.

[0095] In some selectable embodiments of the present disclosure, the second decision unit 23 is configured to determine a second similarity between a traffic object of a first category in the image of each frame of the plurality of frame images and a template image of each second category, where each second category is a subcategory of the first category, and to determine, based on the second similarity, the subcategory with the highest confidence level for the traffic object of the first category in the image of each frame and the confidence level of the subcategory.

[0096] In some selectable embodiments of the present disclosure, the third decision unit 24 is configured to set the subdivision category with the highest confidence level for a traffic object of the first category in the second image to the same category as the subdivision category with the highest confidence level for a traffic object of the first category in the first image, in response to the fact that the confidence level of the subdivision category with the highest confidence level for a traffic object of the first category in the first image of the plurality of frame images satisfies a third preset condition, and the confidence level of the subdivision category with the highest confidence level for a traffic object of the first category in the second image of the plurality of frame images does not satisfy the third preset condition.

[0097] In some selectable embodiments of the present disclosure, the third decision unit 24 is configured to output hint information indicating that it cannot determine the classification result of the traffic object in response to the fact that the confidence level of the traffic object in the first category in the plurality of frame images does not satisfy any of the confidence levels of the sub-categorization categories with the highest confidence level of any of the third preset conditions.

[0098] In the embodiments of this disclosure, the acquisition unit 21, the first decision unit 22, the second decision unit 23, and the third decision unit 24 within the apparatus may all be implemented as a central processing unit (CPU), a digital signal processor (DSP), a microcontroller unit (MCU), or a field-programmable gate array (FPGA) in actual applications.

[0099] It should be explained that, when the image processing apparatus provided in the above embodiment performs image processing, only the divisions of each program module described above were given as examples. However, in actual application, the above processing can be completed by different program modules as needed; that is, the internal structure of the apparatus can be divided into different program modules to complete all or part of the processing described above. Furthermore, the embodiments of the image processing apparatus and image processing method provided in the above embodiment belong to the same concept, and their implementation process can be referred to the embodiment of the method.

[0100] Embodiments of this disclosure further provide electronic devices. Figure 6 is a schematic structural diagram of the hardware configuration of an electronic device according to an embodiment of this disclosure. As shown in Figure 6, the electronic device includes a memory 32, a processor 31, and a computer program stored in the memory 32 and executable on the processor 31, and when the processor 31 executes the program, it realizes the steps of the image processing method described in an embodiment of this disclosure.

[0101] Optionally, the electronic device may further include a user interface 33 and a network interface 34. Here, the user interface 33 may include a display, keyboard, mouse, trackball, click wheel, keys, buttons, touchpad, or touchscreen.

[0102] Optionally, each component within the electronic device is coupled together by a bus system 35. It can be understood that the bus system 35 is used to enable communication between these components. In addition to the data bus, the bus system 35 also includes a power bus, a control bus, and a status signal bus. However, for clarity, all the various buses are represented as the bus system 35 in Figure 6.

[0103] Understandably, memory 32 may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. Here, non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disk, or compact disc read-only memory (CD-ROM). Magnetic surface memory may be magnetic disk memory or magnetic tape memory. Volatile memory may be random access memory (RAM) used as an external cache.By illustrative rather than limiting description, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchlink dynamic random access memory (SLDRAM), and direct Rambus random access memory (DRRAM). The memory 32 described in the embodiments of this disclosure includes, but is not limited to, these and any other suitable types of memory.

[0104] The methods disclosed in the embodiments of this disclosure described above may be applied to or implemented by a processor 31. The processor 31 may be an integrated circuit chip with signal processing capabilities. In the process of implementation, each step of the above method may be completed by instructions in the form of hardware integrated logic circuits or software in the processor 31. The processor 31 may be a general-purpose processor, a DSP, or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, etc. The processor 31 may implement or execute each method, step and logic block diagram disclosed in the embodiments of this disclosure. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this disclosure may be implemented and completed by a hardware decoding processor, or directly implemented and completed by a combination of hardware and software modules in the decoding processor. The software modules may reside in a storage medium, which may be located in memory 32, and the processor 31 reads information in memory 32 and, in combination with its hardware, completes the steps of the above method.

[0105] In exemplary embodiments, the electronic device may be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), FPGAs, general-purpose processors, controllers, MCUs, microprocessors, or other electronic components to perform the methods described above.

[0106] In exemplary embodiments, embodiments of the present disclosure further provide a computer-readable storage medium, for example, a memory 32 containing a computer program, the computer program being executed by a processor 31 of an electronic device to complete the steps described above. The computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disk, or CD-ROM, and may be a variety of devices containing one or any combination thereof of the above memories.

[0107] A computer-readable storage medium provided by an embodiment of the present disclosure stores a computer program, and when the program is executed by a processor, it enables the steps of the image processing method described in the embodiment of the present disclosure.

[0108] Embodiments of the present disclosure provide a computer program product comprising a non-temporary computer-readable storage medium on which the computer program is stored, and which, when the computer program is read and executed by a computer, implements some or all of the steps in the image processing method described in the embodiments of the present disclosure.

[0109] The methods disclosed in some of the method embodiments provided in this application can be arbitrarily combined, provided they do not conflict, to obtain new method embodiments.

[0110] The features disclosed in some of the product embodiments provided in this application can be arbitrarily combined, provided they do not contradict each other, to obtain new product embodiments.

[0111] The features disclosed in some of the method or apparatus embodiments provided in this application can be arbitrarily combined, provided they do not conflict, to obtain new method or apparatus embodiments.

[0112] In some embodiments provided herein, it should be understood that the disclosed devices and methods may be implemented in other ways. The embodiments of the devices described above are illustrative only, and for example, the division of the units is only logical and functional, and other divisional methods may be possible in actual implementation, for example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not performed. Furthermore, the interconnection, direct connection or communication connection of each component shown or discussed may be an indirect connection or communication connection through some interface, device or unit, and may be in electrical, mechanical or other forms.

[0113] The units described as separation members may or may not be physically separate, and the members indicated as units may or may not be physical units; that is, they may be located in one place or distributed among multiple network units. Depending on the actual needs, some or all of these units can be selected to achieve the objectives of the solution of this embodiment.

[0114] Furthermore, each functional unit in each embodiment of this disclosure may be integrated into a single processing unit, each unit may be a single unit on its own, or two or more units may be integrated into a single unit. The integrated unit may be implemented in hardware form, or in the form of a hardware and software functional unit.

[0115] As a person skilled in the art will understand, all or some of the steps of implementing the embodiment of the above method may be completed by instructing the relevant hardware by program, and the aforementioned program may be stored in a computer-readable storage medium, and when the program is executed, it performs the steps including the embodiment of the above method, and the aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0116] Alternatively, if the integrated units of the present disclosure are implemented in the form of software function modules and sold or used as independent products, they may be stored on a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of the present disclosure may be essentially or in part contribute to the prior art, embodied in the form of a software product, which is stored on a storage medium and contains several instructions for causing a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods of each embodiment of the present disclosure. The storage medium includes a variety of media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0117] The above descriptions are merely specific embodiments of the Disclosure, and the scope of protection of the Disclosure is not limited thereto. Any changes or substitutions that any person skilled in the art could easily conceive within the technical scope disclosed herein should be included within the scope of protection of the Disclosure. [Industrial applicability]

[0118] Embodiments of this disclosure disclose an image processing method and apparatus, electronic equipment, storage medium, and computer program product. The method includes acquiring a video stream collected by an image acquisition device installed on a vehicle, determining a plurality of frame images containing a specific traffic object from the video stream, determining the category of a specific traffic object in each frame of the plurality of frame images and the confidence level of the category, and determining correction information for categories whose confidence level does not meet a predetermined condition based on a comparison of the confidence levels of the categories of specific traffic objects in the plurality of frame images. In embodiments of this disclosure, by correcting categories whose confidence level does not meet a predetermined condition based on a comparison of the confidence levels of the categories of specific traffic objects in the plurality of frame images, i.e., correcting low-confidence classification results with high-confidence classification results for a specific traffic object, the accuracy of classification of traffic objects in images can be improved on the one hand, and a basis for confidence in downstream policy decision control can be provided on the other hand, facilitating subsequent real-time control.

Claims

1. An image processing method performed by an electronic device, The process involves acquiring a video stream collected by an image acquisition device installed on the vehicle, determining multiple frame images containing a specific traffic object from the video stream, and To determine the category of a specific traffic object in each frame of the aforementioned plurality of frame images and the confidence level of said category, This includes determining correction information for categories whose reliability does not meet pre-set conditions, based on the results of comparing the reliability of specific traffic object categories within the aforementioned multiple frame images. Determining the category of a specific traffic object in each frame of the aforementioned plurality of frame images and the confidence level of said category is: The process involves determining the subdivision category with the highest confidence level for the traffic object in the first category within each frame of the plurality of frame images, and the confidence level of the subdivision category, wherein the first category is the category to which the specific traffic object belongs. Based on the comparison of the confidence levels of specific traffic object categories within the aforementioned multiple frame images, determining the correction information for categories whose confidence levels do not meet the pre-set conditions is: This includes determining correction information for the sub-categories of traffic targets in the first category whose maximum confidence level does not meet a predetermined condition, based on the results of comparing the confidence levels of the sub-categories with the highest confidence level of traffic targets in the multiple frame images. Image processing methods.

2. Determining multiple frame images containing a specific traffic object from the aforementioned video stream is: Determining a first region in the image of a traffic object in an image that includes a traffic object in the aforementioned video stream, For each first region, a second region is determined within the first region, wherein the second region is smaller than the first region. Based on information from each second region, the method involves selecting images containing traffic objects of a first category from images containing the traffic object, wherein the first category is the category to which the specific traffic object belongs. The image processing method according to claim 1.

3. The aforementioned second region is the central region of the aforementioned first region, and the information of the aforementioned second region is the information of the central region of the feature map of the aforementioned first region. The image processing method according to claim 2.

4. Based on the information of each of the second regions, selecting an image containing a traffic object of the first category from an image containing the traffic object is: Feature extraction is performed on each of the second regions, and based on the extracted features, the first similarity of the pixel points in each of the second regions is determined. The method includes determining an image containing a traffic object of the first category as an image in which a second region is located, where the location information satisfies a first pre-set condition and the first similarity satisfies a second pre-set condition. The image processing method according to claim 2.

5. Determining the subdivision category with the highest confidence level for the traffic object of the first category in each frame image among the multiple frame images, and the confidence level of the said subdivision category, is: The process involves determining the second similarity between the traffic object of the first category and the template image of each second category within the image of each frame among the plurality of frame images, wherein each second category is a subcategory of the first category. This includes determining, based on the second similarity, the subdivision category with the highest confidence level for the traffic object of the first category in the image of each frame, and the confidence level of the subdivision category, The image processing method according to claim 1.

6. Based on the comparison of the reliability of the subdivision category with the highest reliability of the traffic object in the first category within the multiple frame images, determining the correction information for the subdivision category of the traffic object in the first category that does not satisfy the pre-set conditions is: In response to the fact that the reliability of the subdivision category with the highest reliability of the traffic object of the first category in the first image among the plurality of frame images satisfies a third pre-set condition, and the reliability of the subdivision category with the highest reliability of the traffic object of the first category in the second image among the plurality of frame images does not satisfy the third pre-set condition, the subdivision category with the highest reliability of the traffic object of the first category in the second image is set to the same category as the subdivision category with the highest reliability of the traffic object of the first category in the first image, The image processing method according to claim 1.

7. The system further includes outputting hint information indicating that the classification result of the traffic object cannot be determined in response to the fact that the confidence level of the first category of traffic object in the plurality of frame images does not satisfy the third pre-set condition of the highest subdivision category. The image processing method according to claim 6.

8. An image processing method performed by an electronic device, The process involves acquiring a video stream collected by an image acquisition device installed on the vehicle, determining multiple frame images containing a specific traffic object from the video stream, and To determine the category of a specific traffic object in each frame of the aforementioned plurality of frame images and the confidence level of said category, This includes determining correction information for categories whose reliability does not meet pre-set conditions, based on the results of comparing the reliability of specific traffic object categories within the aforementioned multiple frame images. Determining the category of a specific traffic object in each frame of the aforementioned plurality of frame images and the confidence level of said category is: The process involves determining the subdivision category with the highest confidence level for the traffic object in the first category within each frame of the plurality of frame images, and the confidence level of the subdivision category, wherein the first category is the category to which the specific traffic object belongs. Determining the subdivision category with the highest confidence level for the traffic object of the first category in each frame image among the multiple frame images, and the confidence level of the said subdivision category, is: The process involves determining the second similarity between the traffic object of the first category and the template image of each second category within the image of each frame among the plurality of frame images, wherein each second category is a subcategory of the first category. This includes determining, based on the second similarity, the subdivision category with the highest confidence level for the traffic object of the first category in the image of each frame, and the confidence level of the subdivision category, Image processing methods.

9. An image processing apparatus comprising an acquisition unit, a first decision unit, a second decision unit, and a third decision unit, The acquisition unit is configured to acquire a video stream collected by an image acquisition device installed on the vehicle. The first decision unit is configured to determine a plurality of frame images containing a specific traffic object from the video stream, The second decision unit is configured to determine the category of a specific traffic object in each frame of the plurality of frame images and the confidence level of the category. Determining the category of a specific traffic object in each frame of the aforementioned plurality of frame images and the confidence level of said category is: The process involves determining the subdivision category with the highest confidence level for the traffic object in the first category within each frame of the plurality of frame images, and the confidence level of the subdivision category, wherein the first category is the category to which the specific traffic object belongs. The third decision unit is configured to determine correction information for categories whose reliability does not meet pre-set conditions, based on the results of comparing the reliability of specific traffic object categories within the plurality of frame images. Based on the comparison of the confidence levels of specific traffic object categories within the aforementioned multiple frame images, determining the correction information for categories whose confidence levels do not meet the pre-set conditions is: This includes determining correction information for the sub-categories of traffic targets in the first category whose maximum confidence level does not meet a predetermined condition, based on the results of comparing the confidence levels of the sub-categories with the highest confidence level of traffic targets in the multiple frame images. Image processing device.

10. An image processing apparatus comprising an acquisition unit, a first decision unit, a second decision unit, and a third decision unit, The acquisition unit is configured to acquire a video stream collected by an image acquisition device installed on the vehicle. The first decision unit is configured to determine a plurality of frame images containing a specific traffic object from the video stream, The second decision unit is configured to determine the category of a specific traffic object in each frame of the plurality of frame images and the confidence level of the category. Determining the category of a specific traffic object in each frame of the aforementioned plurality of frame images and the confidence level of said category is: The process involves determining the subdivision category with the highest confidence level for the traffic object in the first category within each frame of the plurality of frame images, and the confidence level of the subdivision category, wherein the first category is the category to which the specific traffic object belongs. Determining the subdivision category with the highest confidence level for the traffic object of the first category in each frame image among the multiple frame images, and the confidence level of the said subdivision category, is: The process involves determining the second similarity between the traffic object of the first category and the template image of each second category within the image of each frame among the plurality of frame images, wherein each second category is a subcategory of the first category. This includes determining, based on the second similarity, the subdivision category with the highest confidence level for the traffic object of the first category in the image of each frame, and the confidence level of the subdivision category, The third decision unit is an image processing device configured to determine correction information for categories whose reliability does not meet pre-set conditions, based on the results of comparing the reliability of categories of specific traffic objects in the plurality of frame images.

11. A computer-readable storage medium, which, when executed by a processor, stores a computer program in the processor for implementing the image processing method described in any one of claims 1 to 8.

12. Electronic device comprising memory, a processor, and a computer program stored in memory and executable on the processor, wherein when the processor executes the program, it implements the image processing method described in any one of claims 1 to 8.

13. A computer program that causes a computer to implement the image processing method described in any one of claims 1 to 8.

Images