IMAGE PROCESSING METHOD, IMAGE PROCESSING APPARATUS, ELECTRONIC DEVICE, COMPUTER-READABLE STORAGE MEDIUM, AND COMPUTER PROGRAM

Abstract

The embodiments of the present disclosure disclose an image processing method and its device, electronic device, storage medium, and computer program product, which includes: acquiring a video stream collected by an image collecting device installed on a traveling device, determining a plurality of frame images including a specific traffic object from the video stream, determining a category of the specific traffic object in each frame image of the plurality of frame images and a reliability of the category, and determining correction information of a category whose reliability does not meet a preset condition based on a comparison result of the reliability of the category of the specific traffic object in the plurality of frame images.

Description

[Technical field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This disclosure is filed based on a Chinese patent application having application number 202210301591.0, filing date March 24, 2022, and title "Image processing method and apparatus, electronic device, and storage medium", and claims priority to the Chinese patent application, the entire contents of which are incorporated herein by reference.

[0002] The present disclosure relates to the field of image processing technology, but is not limited thereto, and specifically to an image processing method and device therefor, an electronic device, a storage medium, and a computer program product. [Background technology]

[0003] In related technology, classification and identification of traffic objects (e.g., traffic signs) can be achieved by a multi-classification task model, but there is a problem that the identification accuracy is not high because there are many traffic objects and they are easily affected by long distances and object occlusion. Summary of the Invention [Means for solving the problem]

[0004] In view of this, embodiments of the present disclosure provide an image processing method and apparatus, an electronic device, a storage medium, and a computer program product.

[0005] The technical solutions of the embodiments of the present disclosure are realized as follows.

[0006] An embodiment of the present disclosure provides an image processing method, the image processing method comprising: obtaining a video stream collected by an image collection device installed on the traveling equipment, and determining a number of frame images from the video stream that include a particular traffic object; determining a category of a particular traffic object in each frame image of the plurality of frame images and a confidence level of the category; and determining correction information for a category whose reliability does not satisfy a preset condition based on a comparison result of reliability of the category of the specific traffic object in the plurality of frame images.

[0007] An embodiment of the present disclosure further provides an image processing apparatus, the image processing apparatus including: an acquisition unit, a first determination unit, a second determination unit and a third determination unit; The acquisition unit is configured to acquire a video stream collected by an image acquisition device installed on a traveling machine; The first determining unit is configured to determine a plurality of frame images including a particular traffic object from the video stream; The second determination unit is configured to determine a confidence level of a category of a particular traffic object in an image of each frame of the plurality of frame images; The third determination unit is configured to determine, based on a comparison result of the reliability of categories of specific traffic objects in the plurality of frame images, modification information of the categories whose reliability does not satisfy a preset condition.

[0008] An embodiment of the present disclosure further provides a computer-readable storage medium storing a computer program which, when executed by a processor, causes the processor to implement the steps of the image processing method according to the embodiment of the present disclosure.

[0009] An embodiment of the present disclosure further provides an electronic device, the electronic device including a memory, a processor and a computer program stored in the memory and executable on the processor, the processor implementing steps of an image processing method described in an embodiment of the present disclosure when executing the program.

[0010] An embodiment of the present disclosure provides a computer program product, the computer program product including a non-transitory computer-readable storage medium having a computer program stored thereon, the computer program being capable of implementing some or all of the steps of the image processing method described in the embodiment of the present disclosure when read and executed by a computer.

[0011] The embodiments of the present disclosure provide an image processing method and its device, electronic device, storage medium, and computer program product. The method includes: acquiring a video stream collected by an image collecting device installed on a traveling device, determining a plurality of frame images including a specific traffic object from the video stream; determining a category of the specific traffic object in each frame image of the plurality of frame images and a reliability of the category; and determining correction information of a category whose reliability does not meet a preset condition based on a comparison result of the reliability of the category of the specific traffic object in the plurality of frame images. By adopting the technical solution of the embodiment of the present disclosure, the category whose reliability does not meet a preset condition is corrected based on a comparison result of the reliability of the category of the specific traffic object in the plurality of frame images, that is, the low-reliability classification result of the specific traffic object is corrected by the high-reliability classification result of the specific traffic object, which on the one hand can improve the classification accuracy of the traffic object in the image, and on the other hand can provide a basis of confidence for downstream policy decision control, and facilitate subsequent real-time control. [Brief description of the drawings]

[0012] [Figure 1] FIG. 2 is a schematic diagram of an application scenario of the image processing method according to an embodiment of the present disclosure. [Diagram 2] 1 is a schematic flow chart of an image processing method according to an embodiment of the present disclosure. [Diagram 3] FIG. 13 is a schematic diagram of a classification result in the image processing method according to the embodiment of the present disclosure. [Figure 4] FIG. 13 is a schematic diagram of the reliability of classification results in the image processing method according to the embodiment of the present disclosure. [Diagram 5] FIG. 1 is a structural schematic diagram of an image processing device according to an embodiment of the present disclosure. [Figure 6] FIG. 2 is a structural schematic diagram of a hardware configuration of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] The embodiments of the present disclosure will now be described in more detail with reference to the drawings and specific examples.

[0014] In the related art, the identification of traffic objects is mainly realized by a single-layer multi-classifier, and there are many categories of traffic objects, and they are easily affected by long distances and object occlusion, etc., which makes labeling difficult and makes it difficult to realize accurate classification. FIG. 1 is a schematic diagram of an application scenario of an image processing method according to an embodiment of the present disclosure. As shown in FIG. 1, for the image of the 21st frame (Frame21), the distance is too far to identify the content of the traffic sign board. For the image of the 51st frame (Frame51), the traffic sign board is easily identified as a speed limit of 30 because it is blocked by a tree trunk. For the image of the 55th frame (Frame55), the traffic sign board is easily identified as a speed limit of 60 because it is similarly blocked by a tree trunk. Finally, in the image of the 60th frame (Frame60), the traffic sign board can be correctly identified as a speed limit of 50.

[0015] In an embodiment of the present disclosure, the electronic device corrects the categories whose reliability does not meet a preset condition based on the comparison result of the reliability of the categories of specific traffic objects in multiple frame images, i.e., corrects the low-reliability classification result of the specific traffic object with the high-reliability classification result of the specific traffic object, which can on the one hand improve the classification accuracy of the traffic objects in the images, and on the other hand provide a reliable basis for downstream policy decision control.

[0016] In each embodiment of the present disclosure, the traffic object may be any object on a road, and may include at least one of the following objects: a traffic sign board, a road sign, a traffic participant, and a traffic light.

[0017] In the embodiments of the present disclosure, it should be explained that the terms "comprises", "has", or any other variations thereof are intended to cover a non-exclusive inclusion, whereby a method or apparatus including a set of elements includes not only the elements explicitly described, but also other elements not explicitly listed, or elements inherent to the method or apparatus implementation. In the absence of further limitations, an element qualified by the phrase "comprises a ..." does not exclude the presence of further related elements in the method or apparatus including the element (e.g., a step in a method or a unit in an apparatus, e.g., a unit may be a part of a circuit, a part of a processor, a part of a program or software, etc.).

[0018] For example, the image processing method provided by the embodiment of the present disclosure includes a series of steps, but the image processing method provided by the embodiment of the present disclosure is not limited to the steps described. Similarly, the image processing device provided by the embodiment of the present disclosure includes a series of modules, but the device provided by the embodiment of the present disclosure is not limited to including clearly recorded modules, and may further include modules that need to be set to obtain related information or when performing processing based on information.

[0019] The term "and / or" as used herein is only for describing the associative relationship of related objects, and indicates that three types of relationships can exist, for example, A and / or B can indicate three situations: A exists alone, A and B exist simultaneously, and B exists alone. Also, the term "at least one" as used herein can indicate any combination of at least two of a plurality of elements, for example, including at least one of A, B, and C can indicate including any one or more elements selected from the set consisting of A, B, and C.

[0020] An embodiment of the present disclosure provides an image processing method. Figure 2 is a schematic flow chart of the image processing method according to an embodiment of the present disclosure. As shown in Figure 2, the method includes steps 101 to 103.

[0021] In step 101, a video stream collected by an image collection device installed on a traveling machine is obtained, and a number of frame images including a specific traffic object are determined from the video stream.

[0022] In step 102, a category of a particular traffic object in each frame image of the plurality of frame images and a confidence level for that category are determined.

[0023] In step 103, based on the comparison result of the reliability of the categories of specific traffic objects in the plurality of frame images, the correction information of the categories whose reliability does not meet the preset condition is determined.

[0024] The image processing method of this embodiment is applied to an electronic device, and the electronic device may be an in-vehicle device, a cloud platform, or other computer devices. For example, the in-vehicle device may be a thin client, a thick client, a microprocessor-based system, a small computer system, etc. installed on a traveling device, and the cloud platform may be a distributed cloud computing technology environment including a small computer system or a large computer system. Here, the traveling device may be, for example, various vehicles traveling on a road. In the following embodiments, the traveling device is described as a vehicle as an example.

[0025] In this embodiment, the in-vehicle device can be communicatively connected to a sensor, a positioning device, etc. of the vehicle, and the in-vehicle device can acquire data collected by the sensor of the vehicle and geographical location information reported by the positioning device through the communicative connection. Exemplarily, the sensor of the vehicle may be at least one of a millimeter wave radar, a laser radar, a camera, etc. The positioning device may be a device for providing a positioning service based on at least one of a Global Positioning System (GPS), a Beidou satellite navigation system, or a Galileo satellite navigation system.

[0026] In one example, the in-vehicle device may be an Advanced Driving Assistant System (ADAS), where the ADAS is installed in the vehicle, and the ADAS may obtain real-time position information of the vehicle from a positioning device of the vehicle, and / or the ADAS may obtain image data, radar data, etc. representing the vehicle's surrounding environment information from a sensor of the vehicle. Here, optionally, the ADAS can transmit vehicle driving data including the vehicle's real-time position information to the cloud platform, and thus the cloud platform can receive the image data, radar data, etc. representing the vehicle's real-time position information and / or the vehicle's surrounding environment information.

[0027] In this embodiment, the video stream is acquired by an image collection device (i.e., the above-mentioned sensor such as a camera) installed on the traveling equipment, and the image collection device collects road images or environmental images around the traveling equipment in real time as the traveling equipment moves. That is, the video stream may be a continuous image obtained by continuously collecting the surrounding environment or scenario in the traveling state by the traveling equipment.

[0028] In some alternative embodiments, the electronic device may use a classification network to identify the image of each frame in the video stream, and determine the specific traffic object included in the image of each frame and the category of the specific traffic object. Exemplarily, the video stream may be input data of the classification network, and the classification network may perform feature extraction on the image of each frame in the video stream, and determine the specific traffic object in the image based on the extracted features, determine a first region of the specific traffic object in the image, and determine the category of the specific traffic object. Here, the category of the specific traffic object may be one category of a classification of multiple traffic objects. For example, each traffic object may be pre-divided into multiple categories, and each category may include one or more traffic objects. The category to which the specific traffic object belongs may be one of the above-mentioned pre-divided categories.

[0029] In some selectable embodiments, determining a confidence level of a particular traffic object category in each frame image of the plurality of frame images may include determining a template image of the particular traffic object category in each frame image of the plurality of frame images, calculating a similarity between the particular traffic object and the template image, and determining a confidence level of the particular traffic object category based on the similarity.

[0030] In this embodiment, a template image of each category is stored in the electronic device, and after determining the category of the specific traffic object, the electronic device can compare the image of the first area where the specific traffic object is located with the template image of the corresponding category, calculate the similarity between the specific traffic object and the template image of the corresponding category, and use the calculated similarity as the confidence of the category of the specific traffic object.

[0031] In this embodiment, the electronic device determines the modification information of the category whose reliability does not satisfy the preset condition based on the comparison result of the reliability of the category of the specific traffic object in the plurality of frame images. That is, the electronic device modifies the category whose reliability does not satisfy the preset condition. Here, for example, the reliability not satisfying the preset condition may be the reliability is smaller than the preset threshold.

[0032] Optionally, determining correction information of a category whose reliability does not satisfy a preset condition based on a comparison result of the reliability of the category of the specific traffic object in the plurality of frame images includes: in response to the reliability of the category of the specific traffic object in a first image of the plurality of frame images satisfying a preset condition and the reliability of the category of the specific traffic object in a second image of the plurality of frame images not satisfying the preset condition, setting the category of the specific traffic object in the second image to the category of the specific traffic object in the first image, where the order of the first image and the second image is not limited, i.e., the first image is after the second image, or the first image is before the second image.

[0033] In this embodiment, the electronic device can use a high-reliability classification result for a specific traffic object to correct a low-reliability classification result for the specific traffic object, realize the correction of the category whose reliability does not meet the preset condition, and provide sufficient basis for downstream modules (e.g., a control module, a policy-making module, etc.), facilitating subsequent real-time control.

[0034] In some optional embodiments of the present disclosure, determining a plurality of frame images including a specific traffic object from the video stream includes: determining a first region in an image of the traffic object in an image including the traffic object in the video stream; and for each first region, determining a second region within the first region, where the second region is smaller than the first region; and based on information of each second region, selecting images including a first category of traffic objects from the images including the traffic object as a plurality of frame images including the specific traffic object, where the first category is a category to which the specific traffic object belongs.

[0035] In this embodiment, the electronic device determines a first region in an image of a traffic object in an image of each frame in the video stream, i.e. obtains a detection frame (e.g. a rectangular frame) in the image of the traffic object, the region in the image of the detection frame being the first region, and further determines a second region within the first region in each image comprised in the video stream.

[0036] Optionally, the second region is a central region of the first region, and the information of the second region is information of a central region of a feature map of the first region.

[0037] For example, the method of determining the second region may be to proportionally reduce the length and width of the first region (i.e., the detection frame of the traffic object), and the region after the reduction may be the second region, and the second region may be called the center region of the first region. For another example, the method of determining the second region may be to non-proportionally reduce the length and width of the first region according to the degree of obstruction of the traffic object, and move the center point according to the obstructed position of the traffic object after the reduction, thereby obtaining the second region. Taking the image of the 51st frame or the image of the 55th frame in FIG. 1 as an example, in order to detect a traffic object and find that the left side of the traffic object is obstructed, the length and width of the first region (i.e., the detection frame of the traffic object) in which the traffic object is located is non-proportionally reduced, and since the left side of the traffic object is obstructed after the reduction, the region after the reduction is moved to the right (the region after the movement is also within the first region), thereby obtaining the second region. In this way, the characteristics of the traffic object are left as much as possible in the second region, and the characteristics of the obstruction are reduced.

[0038] In this embodiment, the electronic device screens images including traffic objects of a first category from images including traffic objects based on information of a second region in the image, and sets the images including traffic objects of the first category in multiple frames as multiple frame images including a specific traffic object, where the first category is a category to which the specific traffic object belongs.

[0039] In some alternative embodiments, selecting images including traffic objects of a first category from the images including the traffic objects based on information of each of the second regions includes: performing feature extraction for each of the second regions respectively, determining a first similarity of pixel points of each of the second regions based on the extracted features, and determining an image in which a second region is located whose position information satisfies a first preset condition and whose first similarity satisfies a second preset condition as an image including a traffic object of the first category, where the first category is a category to which a specific traffic object belongs.

[0040] Here, optionally, the location information of the second region in the image containing the traffic object satisfies a first preset condition may be that the distance between the location information of the second region in the image of any two adjacent frames containing the traffic object, i.e., the difference between the locations of the second region in the located images, is smaller than a first threshold, where the distance may be a distance in a specified coordinate system (e.g., a pixel coordinate system, an image coordinate system, etc.).

[0041] Optionally, the first similarity satisfying a second predefined condition may refer to the first similarity being greater than or equal to a second threshold.

[0042] Exemplarily, the image including the traffic object includes a first image and a second image, and the second image is an image of a frame after the first image. In one example, the second image may be an image of a frame after the first image. In another example, the second image may be an image several frames after the first image. For example, in FIG. 1, the first image may be an image of the 21st frame, and the second image may be an image of the 51st frame, the 55th frame, or the 60th frame.

[0043] The electronic device performs traffic object identification on the first image and the second image, respectively, and determines a first region of the traffic object in the first image and a first region of the traffic object in the second image. In this example, tracking the identified traffic object in the first image is performed using a second region smaller than the first region in which the traffic object is located, and considering an occlusion situation, since the edge of the traffic object is usually occluded, tracking using the second region is robust against occlusion.

[0044] For example, the length and width of the first region (i.e., detection window) may be proportionally reduced, and the reduced region may be a second region, which may be referred to as a central region of the first region. In some embodiments, based on the pixel points of the second region in the first image and the pixel points of the second region in the second image, it may be determined whether a traffic object in the first image and the second image is a traffic object of a first category.

[0045] In some embodiments, a feature extraction process may be performed on pixel points of the second region in the first image, and a feature extraction process may be performed on pixel points of the second region in the second image, and a similarity between the two (herein referred to as a first similarity) may be calculated based on the extracted features. If the first location information and the second location information satisfy a first preset condition, and the first similarity satisfies a second preset condition, it is determined that the traffic objects corresponding to the first image and the second region in the second image are of a first category. Here, the first location information may be the coordinates of a center point of the second region in the first image, and the second location information may be the coordinates of a center point of the second region in the second image.

[0046] Here, optionally, the first location information and the second location information satisfying the first preset condition may be that the distance between the first location information and the second location information (e.g., the distance between the center point coordinate of the second area in the first image and the center point coordinate of the second area in the second image) is smaller than a first threshold. Here, the distance may be a distance in a specified coordinate system (e.g., a pixel coordinate system, an image coordinate system, etc.). For example, in the pixel coordinate system, a first center point coordinate corresponding to the first location information is determined, a second center point coordinate corresponding to the second location information is determined, and the first center point coordinate and the second center point coordinate are subtracted to obtain the distance between the first location information and the second location information. In practical application, since the frame interval of image collection is very small, if a traffic object of the first category is included in different frame images, the positions of the traffic objects in the different frame images are also close.

[0047] Optionally, the first similarity satisfying a second preset condition may be the first similarity being equal to or greater than a second threshold. If the first similarity is equal to or greater than a second threshold, it may be determined that the traffic object in the second region in the first image and the traffic object in the second region in the second image are of the same category, i.e., objects of the first category (but may not be the same traffic object). Then, by combining the above-mentioned first location information and the second location information satisfying the first preset condition, it may be determined that the traffic object corresponding to the second region in the first image and the traffic object corresponding to the second region in the second image are the same traffic object.

[0048] In one case, feature extraction is performed on the second region in each image, and a first similarity of pixel points of the second region in each image can be determined based on the extracted features. If the first similarity satisfies a second preset condition, it can be determined that the traffic objects in each image are of the same category (e.g., the first category). In such a case, it can be determined that the traffic objects in each image are of the same category (e.g., the first category) but may not be the same traffic object. In another case, based on determining the above first similarity, if the position information of the second region in each image satisfies a first preset condition and the first similarity satisfies a second preset condition, it can be determined that the traffic objects in each image are of the same category (e.g., the first category) and are the same traffic object. That is, the traffic objects of the first category described in this embodiment are not limited to the category to which the traffic objects in at least two frames of images belong being the same, but can also include a situation in which the traffic objects in at least two frames of images are the same traffic object.

[0049] In implementation, the fact that a traffic object in at least two frame images is the same traffic object can be determined by the following solution: A unique identifier (ID) can be assigned to each identified traffic object in each frame image so as to facilitate identifying the same traffic object among different frame images, and in response to a situation where traffic objects in multiple frame images are of the same category (e.g., a first category), the first identifiers assigned to the traffic objects in the multiple frame images are associated.

[0050] In this embodiment, taking the multiple frame images as an example including a first image and a second image, after identifying a traffic object (e.g., denoted as object 1) in the first image, a first identifier is assigned to the traffic object (object 1), and when it is determined that a traffic object (e.g., denoted as object 2) corresponding to the second region in the second image and a traffic object (object 1) corresponding to the second region in the first image belong to the same category (e.g., the first identifier assigned to object 1 may be associated with object 2, i.e., both object 1 and object 2 are associated with the first identifier, and therefore are traffic objects of the same category (e.g., the first category). Or, when it is determined that a first identifier is assigned to object 1 and a second identifier is assigned to object 2, and both object 1 and object 2 belong to the same category (e.g., the first category), the first identifier and the second identifier are associated, for example, the second identifier may be replaced with the first identifier, i.e., both object 1 and object 2 are associated with the first identifier, and therefore are traffic objects of the same category (e.g., the first category).

[0051] In some optional embodiments of the present disclosure, determining a category and a confidence level of the category of a particular traffic object in each frame image of the plurality of frame images includes determining a sub-category and a confidence level of the sub-category of a first category of traffic object in each frame image of the plurality of frame images having a highest confidence level, the first category being a category to which the particular traffic object belongs.

[0052] Correspondingly, determining correction information of a category whose reliability does not satisfy a preset condition based on a comparison result of the reliability of the category of a specific traffic object in the multiple frame images includes determining correction information of a subdivision category of a first category of traffic object whose maximum reliability does not satisfy a preset condition based on a comparison result of the reliability of a subdivision category of the first category of traffic object in the multiple frame images whose reliability is maximum.

[0053] In this embodiment, the electronic device first determines a first category of traffic objects (i.e., a rough classification category), and then determines a subcategory of the traffic objects in the first category, i.e., first performs coarse-grained classification on the traffic objects, and then performs fine-grained classification on the traffic objects in the coarse-grained classification.

[0054] Optionally, the electronic device can identify the image of each frame in the video stream through a first layer network, and determine that the image of each frame contains traffic objects of the same category (first category), i.e. detect the traffic objects in the image of each frame, and determine that the detected traffic objects belong to the same category (i.e., first category). Exemplarily, the video stream can be input data of the first layer network, and perform feature extraction on the image of each frame in the video stream through the first layer network, determine the traffic objects in the image of each frame based on the extracted features, determine a first region of the traffic objects in the image of each frame, and determine a category (e.g., a rough classification category) of the traffic objects, i.e. output the detection frame of the traffic objects in the image of each frame and the category (rough classification category) to which the traffic objects belong, and further determine a plurality of frame images of traffic objects belonging to the same category (herein referred to as the first category). Here, optionally, the plurality of frame images can be consecutive or non-consecutive frame images in the video stream. For example, the video stream includes 100 frame images, and the determined plurality of frame images including the first category traffic object may be the 10th to 50th frame images, or the 5th, 15th, 25th, 35th, 45th frame images, etc., of the 100 frame images, and is not limited in this embodiment.

[0055] In some alternative embodiments, the category (including the first category) to which the specific traffic object belongs is one of a plurality of traffic object classifications. It can be understood that the first layer network is pre-trained and obtained based on the traffic object classifications, and the processing of the first layer network on an image can obtain whether and which traffic object in the image belongs to the pre-labeled traffic object classification.

[0056] For example, take the traffic object as a traffic sign (including, for example, a traffic sign board and a road sign) as an example, since there are many categories of traffic signs, in this embodiment, various traffic signs are pre-classified. For example, as shown in FIG. 3, traffic signs can be pre-classified into a number of first categories 41, such as speed class signs, sidewalk class signs, warning class signs, and stop class signs. Assuming that a traffic object in an image of each frame in a video stream is identified, a number of frame images including a traffic object in the category of "speed class signs" can be screened. In practical applications, each traffic sign can be classified according to its function or action. In other embodiments, other classification methods can also be used, but are not limited in this embodiment.

[0057] In some embodiments, after a plurality of frame images including traffic objects of the same category (i.e., a first category) are determined via a first layer network, a sub-classification process is performed on the traffic objects of the first category in each frame image among the plurality of frame images via a second layer network, and sub-categorization categories of the traffic objects of the first category in each frame image and confidence levels of the sub-categories are obtained.

[0058] In this embodiment, the second layer network may be a classification network corresponding to the category to which the traffic object belongs. Optionally, the number of second layer networks may correspond to the number of categories to which the traffic object belongs, i.e., each category to which each traffic object belongs may correspond to one second layer network, and each second layer network is pre-labeled with a subcategory in the category to which the corresponding traffic object belongs. Taking the speed class sign shown in FIG. 3 as an example, the speed class sign may include multiple subcategories 42, such as an 80 kilometer per hour (km / h) speed sign, a 40 km / h speed sign, a 120 km / h speed sign, a 70 km / h speed sign, etc. After determining that the traffic object is a speed class sign, a classification process of the corresponding second layer network can obtain the subcategory with the highest confidence of the traffic object and the confidence of the subcategory. For example, the subcategory with the highest confidence of the traffic object may be a 70 km / h speed sign.

[0059] In another embodiment, the second layer network may correspond to a category to which multiple traffic objects belong. Taking the traffic object as a traffic sign as an example, the second layer network may be used to identify subcategories in "One class sign", "Turn class sign" and "Lane class sign". Or, the second layer network may include multiple branch networks for classification processing, and each branch network may be used to identify subcategories corresponding to categories to which one or more classes of traffic objects belong. Exemplarily, after identifying a category to which the traffic object belongs (e.g., a first category), the electronic device cuts out a sub-image corresponding to a first area in which the traffic object is located, and the sub-image is input to a branch network corresponding to the first category to be used for identifying a subcategory in the first category.

[0060] In this way, a first category (rough classification category) of the traffic object is first determined, and then a subcategory of the traffic object in the first category is determined; that is, a coarse-grained classification is first performed on the traffic object, and then a fine-grained classification is performed on the traffic object in the coarse-grained classification. This can improve the classification accuracy of traffic objects in an image (e.g., traffic signs, road signs, etc.). In particular, when identifying traffic objects with many categories using a single-layer multi-classifier, the current problem of difficulty in labeling due to the large number of categories and inability to classify accurately can be improved.

[0061] In some optional embodiments of the present disclosure, determining a subdivision category with a highest confidence for a first category traffic object in each frame image of the plurality of frame images and a confidence level for the subdivision category includes determining a second similarity between the first category traffic object in each frame image of the plurality of frame images and a template image of each second category, each second category being a subdivision category of the first category, and determining a subdivision category with a highest confidence for the first category traffic object in each frame image of the plurality of frame images and a confidence level for the subdivision category based on the second similarity.

[0062] In this embodiment, the template image of each second category is stored in the electronic device. After the electronic device determines the first category of the traffic object, the electronic device compares the image (which may be a feature map of the area where the traffic object is located) with the template image of each second category, and determines a similarity (herein referred to as the second similarity) between the traffic object of the first category in the image of each frame and the template image of each second category. Here, the maximum second similarity can be the confidence of the subcategory (e.g., the second category) of the traffic object, or the confidence of the subcategory (e.g., the second category) of the traffic object can be calculated based on the maximum second similarity.

[0063] For example, as shown in FIG. 4, a traffic object in an image of a 60th frame (Frame60) is compared with a template image of each second category, and a second similarity to a "50km / h speed sign" is determined to be 100%. A traffic object in an image of a 55th frame (Frame55) is compared with a template image of each second category, and a second similarity to a "60km / h speed sign" is determined to be 50%. A traffic object in an image of a 51st frame (Frame51) is compared with a template image of each second category, and a second similarity to a "30km / h speed sign" is determined to be 40%. A traffic object in an image of a 21st frame (Frame21) is compared with a template image of each second category, and a second similarity to a "no U-turn sign" is determined to be 80%. A subcategory with the highest reliability and the reliability of the subcategory are determined from each second similarity. In the above example, the highest reliability is 100%, and the corresponding subcategory is "50km / h speed sign".

[0064] Exemplarily, if the maximum confidence corresponding to the second category to which the traffic object belongs in the plurality of frame images is equal to or greater than a third threshold, it is determined that the confidence (i.e., reliability) of the second category to which the traffic object belongs is high. Correspondingly, if the maximum confidence corresponding to the second category to which the traffic object belongs in the plurality of frame images is smaller than the third threshold, it is determined that the confidence (i.e., reliability) of the second category to which the traffic object belongs is low. As shown in FIG. 4, the maximum confidence corresponding to the second category to which the traffic object belongs in the image of the 60th frame (Frame60) is 100%, which is regarded as high confidence. In the image of the 21st frame (Frame21), the maximum confidence corresponding to the second category to which the traffic object belongs is 80%, in the image of the 51st frame (Frame51), the maximum confidence corresponding to the second category to which the traffic object belongs is 40%, and in the image of the 55th frame (Frame55), the maximum confidence corresponding to the second category to which the traffic object belongs is 50%, which may all be regarded as low confidence.

[0065] Here, it should be explained that the above third threshold value can be determined based on the actual situation. As an embodiment, the third threshold value can be determined based on the maximum reliability corresponding to the image of each frame. For example, in 100 frames of images, if the images with a maximum reliability of 100% account for 80%, the images with a maximum reliability of 80% account for 15%, and the images with a maximum reliability of 50% account for 5%, the classification result of the image is considered to be relatively reliable, and the value of the third threshold value can be set relatively high, for example, 90% to 95%. Correspondingly, if the overall calculation result of the maximum reliability in the image is not high, the value of the third threshold value can be set small. This embodiment is not limited.

[0066] In this embodiment, the electronic device determines, based on the comparison result of the reliability of the subcategory with the maximum reliability of the first category traffic object in the multiple frame images, correction information of the subcategory of the first category traffic object whose maximum reliability does not meet the preset condition, thereby correcting the subcategory of the first category traffic object whose maximum reliability does not meet the preset condition, further improving the classification accuracy of traffic objects in the video stream, and providing a reliable basis for downstream policy decision control.

[0067] In some optional embodiments of the present disclosure, determining correction information for the subdivision category of the first category traffic object whose maximum confidence does not satisfy a preset condition based on the comparison result of the confidence of the subdivision category with the highest confidence of the first category traffic object in the plurality of frame images includes, in response to the confidence of the subdivision category with the highest confidence of the first category traffic object in a first image of the plurality of frame images satisfying a third preset condition and the confidence of the subdivision category with the highest confidence of the first category traffic object in a second image of the plurality of frame images not satisfying the third preset condition, setting the subdivision category with the highest confidence of the first category traffic object in the second image to the same category as the subdivision category with the highest confidence of the first category traffic object in the first image.

[0068] Here, for example, the reliability satisfying the third preset condition may be the reliability being equal to or greater than a fourth threshold.

[0069] In this embodiment, the order of the first and second images among the multiple frame images is not limited, i.e., the first image is after the second image, or the first image is before the second image.

[0070] In some alternative embodiments, the second image is an image of a frame after the first image. This embodiment is applied to a scenario in which real-time detection is performed on an image.

[0071] Exemplarily, the reliability degree satisfying the third preset condition may indicate high reliability or high reliability. Correspondingly, the reliability degree not satisfying the third preset condition may indicate low reliability or low reliability. For traffic objects of the same category (first category), if the reliability degree of the largest second category (i.e., subdivision category) of the traffic object in the first image is high (satisfies the third preset condition) and the reliability degree of the largest second category (i.e., subdivision category) of the traffic object in the second image is low (does not meet the third preset condition), the high reliability subclassification result (i.e., the largest second category) in the first image can be used to replace the low reliability subclassification result (i.e., the largest second category) in the second image.

[0072] The above embodiment is applied to the real-time detection and classification process for images, such as the classification results of later images are not accurate due to the existence of occlusion. For example, Table 1 is the classification representation before modification, and Table 2 is the classification representation after modification. Referring to Table 1, the real-time detection and classification for the images of 5 frames obtains four traffic objects with identifier IDs of 1, 2, 3, and 4 respectively (here, the traffic objects in the images of the previous two frames are the same traffic objects, so the same IDs are used), the first classification and the second classification of each traffic object, and the corresponding confidence level. If the four traffic objects are determined to be the same traffic object by tracking the traffic objects, the four traffic objects are associated with the same identifier ID "1", and the high-confidence subclassification results are used to replace the low-confidence subclassification results. As shown in Table 2, all the low-confidence subclassification results are replaced with the high-confidence subclassification results, which means that the first classification is speed, the second classification is 50, and the confidence level is high.

[0073] [Table 1]

[0074] [Table 2]

[0075] In this embodiment, the highly reliable subclassification result is used to correct the less reliable subclassification result (the subdivision category with the highest reliability) of the same traffic object, thereby realizing the correction of the subdivision category of the first category traffic object whose maximum reliability does not meet the preset condition, which can provide sufficient basis for downstream modules (e.g., control module, policy decision module, etc.), and facilitate subsequent real-time control.

[0076] In some alternative embodiments, the first image is an image of a frame after the second image. This embodiment applies to scenarios where training data is misclassified.

[0077] In this embodiment, the above images (including the first image and the second image) and classification results are all used to train the network model. When there is a classification error due to long distance, for example, in the scenario shown in FIG. 1, the image of the 21st frame (Frame21) is far away, so the identified subclassification result is low reliability (or confidence level), and as we gradually approach the traffic object, the corresponding collected image becomes clearer and clearer, and the confidence of the subclassification result also changes. For example, the classification result of the 60th frame (Frame60) is high reliability (or confidence level), and the subclassification result of the previous low reliability second image can be replaced by the subclassification result of the later high reliability first image to optimize the training data.

[0078] For example, Table 3 is the classification representation before modification, and Table 4 is the classification representation after modification. Referring to Table 3, through real-time detection and classification for five frames of images, four traffic objects with identifier IDs of 1, 2, 3, and 4, respectively (here, the traffic objects in the previous two frames of images are the same traffic objects, so the same IDs are used), the first classification and the second classification, and the corresponding confidence level of each traffic object are obtained. If the four traffic objects are determined to be the same traffic object by tracking the traffic objects, the four traffic objects are associated with the same identifier ID "1", as shown in Table 4. For traffic objects of the same category (first category), the confidence level of the subclassification results of the previous few frames of images is low (does not meet the third preset condition), and the confidence level of the subclassification results of the same traffic object in the fifth frame of images is high (meets the third preset condition), so that the low confidence subclassification results corresponding to the previous few frames are all replaced with high confidence subclassification results, and the replaced subclassification results are that the first classification is speed, the second classification is 50, and the confidence level is high.

[0079] [Table 3]

[0080] [Table 4]

[0081] In some optional embodiments of the present disclosure, determining correction information for a subdivision category of a first category traffic object whose maximum reliability does not satisfy a preset condition based on a comparison result of reliability of the subdivision category with the maximum reliability of the traffic object of the first category in the plurality of frame images may further include outputting hint information indicating that the classification result of the traffic object cannot be determined in response to any of the reliability of the subdivision category with the maximum reliability of the traffic object of the first category in the plurality of frame images not satisfying a third preset condition.

[0082] In this embodiment, if the reliability of the subcategory with the highest reliability of the first category traffic object in each frame image among the multiple frame images does not satisfy the third preset condition, i.e., the reliability of the subcategory with the highest reliability of the first category traffic object in each frame image is smaller than the fourth threshold, that is, the reliability of the subcategory with the highest reliability of the first category traffic object in each frame image is relatively low, the subcategory of the traffic object cannot be determined, and hint information is outputted, indicating that the classification result cannot be determined.

[0083] For example, referring to the classification results in Table 3 above, through real-time detection and classification for five frames of images, four traffic objects with identifier IDs 1, 2, 3, and 4, respectively (here, the traffic objects in the previous two frames of images are the same traffic objects, so the same IDs are used), the first and second classifications of each traffic object, and the corresponding confidence level are obtained. If the four traffic objects are determined to be the same traffic object by tracking the traffic objects, the four traffic objects are associated with the same identifier ID "1", as shown in Table 5. Assuming that the fourth frame of images is currently collected, the classification results of the second classification corresponding to the same traffic object in the previous three frames of images are all low confidence, and the classification results of the second classification corresponding to the same traffic object in the fourth frame of images are still low confidence (do not meet the third preset condition), so output hint information indicating that the classification result cannot be determined. In some embodiments, as the fifth frame image is collected, if it is detected that the classification result of the second classification corresponding to the same traffic object in the fifth frame image is highly reliable (satisfies the third preset condition), a highly reliable classification result, i.e., a sub-classification result with a speed of 50 km / h, can be output.

[0084] [Table 5]

[0085] Based on the above method embodiment, an embodiment of the present disclosure further provides an image processing device. Figure 5 is a structural schematic diagram of an image processing device according to an embodiment of the present disclosure. As shown in Figure 5, the device includes an acquisition unit 21, a first determination unit 22, a second determination unit 23 and a third determination unit 24.

[0086] Said acquisition unit 21 is adapted to acquire a video stream collected by an image acquisition device installed on the mobile equipment.

[0087] The first determining unit 22 is configured to determine a number of frame images from the video stream that include a particular traffic object.

[0088] The second determination unit 23 is configured to determine a category of a particular traffic object in each frame image of the plurality of frame images and a confidence level of the category.

[0089] The third determination unit 24 is configured to determine, based on a comparison result of the reliability of categories of specific traffic objects in the plurality of frame images, modification information of the categories whose reliability does not satisfy a preset condition.

[0090] In some optional embodiments of the present disclosure, the first determination unit 22 is configured to determine a first region in an image of a traffic object in an image including a traffic object in the video stream, and for each first region, determine a second region within the first region, the second region being smaller than the first region, and based on information of each second region, select an image including a traffic object of a first category from the images including the traffic objects, and make the multi-frame images including traffic objects of the first category into a multi-frame image including a specific traffic object, wherein the first category is a category to which the specific traffic object belongs.

[0091] In some optional embodiments of the present disclosure, the second region is a central region of the first region, and the information of the second region is information of a central region of a feature map of the first region.

[0092] In some optional embodiments of the present disclosure, the first determination unit 22 is configured to respectively perform feature extraction for each of the second regions, determine a first similarity of pixel points of each of the second regions based on the extracted features, and determine an image in which a second region is located whose location information satisfies a first preset condition and whose first similarity satisfies a second preset condition, is an image containing a traffic object of a first category.

[0093] In some optional embodiments of the present disclosure, the second determination unit 23 is configured to determine a subdivision category with the highest reliability of a first category of traffic object in each frame image of the plurality of frame images and the reliability of the subdivision category, the first category being the category to which the particular traffic object belongs.

[0094] In some optional embodiments of the present disclosure, the third determination unit 24 is configured to determine correction information of a subdivision category of a first category traffic object whose maximum reliability does not satisfy a preset condition based on a comparison result of the reliability of the subdivision category of the first category traffic object in the multiple frame images whose maximum reliability is the highest.

[0095] In some optional embodiments of the present disclosure, the second determination unit 23 is configured to determine a second similarity between a first category traffic object in each frame image of the plurality of frame images and a template image of each second category, each second category being a subcategory of the first category, and to determine a subcategory having the highest confidence of the first category traffic object in each frame image and a confidence of the subcategory based on the second similarity.

[0096] In some optional embodiments of the present disclosure, the third determination unit 24 is configured to, in response to the confidence level of a subdivision category with the highest confidence level of a first category traffic object in a first image of the plurality of frame images satisfying a third preset condition and the confidence level of a subdivision category with the highest confidence level of a first category traffic object in a second image of the plurality of frame images not satisfying the third preset condition, set the subdivision category with the highest confidence level of a first category traffic object in the second image to the same category as the subdivision category with the highest confidence level of a first category traffic object in the first image.

[0097] In some optional embodiments of the present disclosure, the third determination unit 24 is configured to output hint information indicating that the classification result of the traffic object cannot be determined in response to any of the reliability of the subdivision categories having the maximum reliability of the traffic object of the first category in the plurality of frame images not satisfying a third preset condition.

[0098] In the embodiment of the present disclosure, the acquisition unit 21, the first determination unit 22, the second determination unit 23 and the third determination unit 24 in the device may all be realized by a central processing unit (CPU), a digital signal processor (DSP), a microcontroller unit (MCU) or a field-programmable gate array (FPGA) in practical applications.

[0099] It should be noted that, when the image processing device provided in the above embodiment performs image processing, only the division of each of the above program modules is taken as an example for description, but in actual application, the above processing can be completed by different program modules as necessary, that is, the internal structure of the device is divided into different program modules to complete all or part of the above-described processing. In addition, the image processing device and the image processing method provided in the above embodiment belong to the same concept, and the implementation process thereof can refer to the method embodiment.

[0100] An embodiment of the present disclosure further provides an electronic device. Figure 6 is a structural schematic diagram of a hardware configuration of an electronic device according to an embodiment of the present disclosure. As shown in Figure 6, the electronic device includes a memory 32, a processor 31, and a computer program stored in the memory 32 and executable on the processor 31, and when the processor 31 executes the program, the electronic device realizes the steps of the image processing method described in the embodiment of the present disclosure.

[0101] Optionally, the electronic device may further include a user interface 33 and a network interface 34. Here, the user interface 33 may include a display, a keyboard, a mouse, a trackball, a click wheel, keys, buttons, a touch pad or a touch screen, etc.

[0102] Optionally, each component in the electronic device is coupled together by a bus system 35. It can be understood that the bus system 35 is used to realize the connection communication between these components. In addition to a data bus, the bus system 35 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the various buses are all labeled as the bus system 35 in FIG. 6.

[0103] As can be appreciated, memory 32 may be volatile or non-volatile memory, or may include both volatile and non-volatile memory. Here, the non-volatile memory may be a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a ferromagnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disk, or a compact disc read-only memory (CD-ROM). The magnetic surface memory may be a magnetic disk memory or a magnetic tape memory. The volatile memory may be a random access memory (RAM) used as an external cache.By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchlink dynamic random access memory (SLDRAM), and direct Rambus random access memory (DRRAM). Memory 32 as described in the embodiments of the present disclosure is intended to include, but is not limited to, these and any other suitable types of memory.

[0104] The methods disclosed in the above embodiments of the present disclosure may be applied to or realized by the processor 31. The processor 31 may be an integrated circuit chip with signal processing capabilities. In the process of realization, each step of the above method may be completed by an instruction in the form of an integrated logic circuit of hardware or software in the processor 31. The processor 31 may be a general-purpose processor, a DSP, or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc. The processor 31 may realize or execute each method, step, and logic block diagram disclosed in the embodiments of the present disclosure. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of the present disclosure may be directly embodied as being executed and completed by a hardware decoding processor, or executed and completed by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium. The storage medium is located in the memory 32, and the processor 31 reads the information in the memory 32 and completes the steps of the above method in combination with the hardware.

[0105] In an exemplary embodiment, the electronic device may be implemented by one or more Application Specific Integrated Circuits (ASICs), DSPs, Programmable Logic Devices (PLDs), Complex Programmable Logic Devices (CPLDs), FPGAs, general purpose processors, controllers, MCUs, microprocessors, or other electronic components to perform the methods described above.

[0106] In an exemplary embodiment, the embodiment of the present disclosure further provides a computer readable storage medium, such as a memory 32 including a computer program, which can be executed by the processor 31 of the electronic device to complete the steps described in the aforementioned method. The computer readable storage medium may be a memory, such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disk, or CD-ROM, or may be various devices including one of the above memories or any combination thereof.

[0107] The computer-readable storage medium provided by the embodiments of the present disclosure stores a computer program, which, when executed by a processor, implements the steps of the image processing method described in the embodiments of the present disclosure.

[0108] An embodiment of the present disclosure provides a computer program product, the computer program product including a non-transitory computer-readable storage medium having a computer program stored thereon, the computer program being capable of implementing some or all of the steps of the image processing method described in the embodiment of the present disclosure when read and executed by a computer.

[0109] The methods disclosed in the several method embodiments provided herein may be combined in any manner, where not inconsistent, to obtain new method embodiments.

[0110] The features disclosed in the several product embodiments provided herein may be combined in any combination, where not inconsistent, to obtain new product embodiments.

[0111] Features disclosed in any method or apparatus embodiment provided herein may be combined in any manner not inconsistent to yield new method or apparatus embodiments.

[0112] In some embodiments provided by the present application, it should be understood that the disclosed devices and methods can be realized in other ways. The above-described device embodiments are only illustrative, for example, the division of the units is only a logical functional division, and other division methods may be used in actual implementation, for example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not implemented. In addition, the mutual coupling or direct coupling or communication connection of each component shown or discussed may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.

[0113] The above units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one place or distributed among multiple network units. Some or all of the units may be selected according to actual needs to achieve the objective of the solution of the present embodiment.

[0114] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, each unit may be a single unit, or two or more units may be integrated into one unit. The integrated unit may be realized in the form of hardware, or in the form of a hardware and software functional unit.

[0115] It can be understood by those skilled in the art that all or part of the steps of implementing the above method embodiments may be completed by instructing relevant hardware by a program, and the above program may be stored in a computer-readable storage medium, which, when executed, performs the steps including the above method embodiments, and the above storage medium includes various media that can store program code, such as mobile storage devices, ROMs, RAMs, magnetic disks or optical disks.

[0116] Or, when the above-mentioned integrated units of the present disclosure are realized in the form of a software functional module and sold or used as an independent product, they may be stored in a computer-readable storage medium. Based on such understanding, the technical solutions of the embodiments of the present disclosure may be essentially or the part that contributes to the prior art may be embodied in the form of a software product, and the computer software product includes some instructions stored in a storage medium to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the method according to each embodiment of the present disclosure. The storage medium includes various media that can store program code, such as a mobile storage device, a ROM, a RAM, a magnetic disk, or an optical disk.

[0117] What has been described above is merely a specific embodiment of the present disclosure, and the scope of protection of the present disclosure is not limited thereto; any changes or replacements that any person skilled in the art can easily think of within the technical scope disclosed in the present disclosure should be included within the scope of protection of the present disclosure. [Industrial Applicability]

[0118] The embodiment of the present disclosure discloses an image processing method and its device, electronic device, storage medium, and computer program product. The method includes: acquiring a video stream collected by an image collecting device installed on a traveling device, determining a plurality of frame images including a specific traffic object from the video stream, determining a category of the specific traffic object in each frame image of the plurality of frame images and a reliability of the category, and determining correction information of a category whose reliability does not meet a preset condition based on a comparison result of the reliability of the category of the specific traffic object in the plurality of frame images. In the embodiment of the present disclosure, based on a comparison result of the reliability of the category of the specific traffic object in the plurality of frame images, the category whose reliability does not meet a preset condition is corrected, i.e., the classification result of the specific traffic object is corrected by the high-reliability classification result, which can improve the classification accuracy of the traffic object in the image on the one hand, and can provide a basis of confidence for downstream policy decision control on the other hand, facilitating subsequent real-time control.

Claims

1. 1. An image processing method, comprising: obtaining a video stream collected by an image collection device installed on the traveling equipment, and determining a number of frame images from the video stream that include a particular traffic object; determining a category of a particular traffic object in each frame image of the plurality of frame images and a confidence level of the category; and determining correction information for a category whose reliability does not satisfy a preset condition based on a comparison result of reliability of the category of a specific traffic object in the plurality of frame images.

2. Determining a plurality of frame images including a particular traffic object from the video stream includes: determining a first region in an image of a traffic object in an image including a traffic object in the video stream; for each first region, determining a second region within the first region, the second region being smaller than the first region; selecting images including traffic objects of a first category as a plurality of frame images including specific traffic objects from the images including the traffic objects based on information of each second region, the first category being a category to which the specific traffic object belongs; The image processing method according to claim 1 .

3. The second region is a central region of the first region, and information on the second region is information on a central region of a feature map of the first region. The image processing method according to claim 2 .

4. selecting, from the images including the traffic object, an image including a traffic object of a first category based on information of each of the second regions; performing feature extraction for each of the second regions, and determining a first similarity between pixel points of each of the second regions based on the extracted features; determining an image in which a second region, the location information of which satisfies a first preset condition and the first similarity satisfies a second preset condition, is located as an image including a traffic object of a first category; The image processing method according to claim 2 .

5. Determining a category of a particular traffic object in each frame image of the plurality of frame images and a confidence level of the category includes: determining a subdivision category having a highest reliability of a first category of traffic object in each frame image of the plurality of frame images and a reliability of the subdivision category, the first category being a category to which the particular traffic object belongs; The image processing method according to claim 1 .

6. Determining correction information for a category whose reliability does not satisfy a preset condition based on a comparison result of the reliability of the category of the specific traffic object in the plurality of frame images, determining correction information for a subcategory of the first category of traffic objects whose maximum reliability does not satisfy a preset condition based on a comparison result of reliability of the subcategory of the first category of traffic objects in the plurality of frame images whose reliability is the maximum; The image processing method according to claim 5.

7. Determining a subdivision category having a highest reliability of a first category of traffic object in each frame image of the plurality of frame images and a reliability of the subdivision category includes: determining a second similarity between a traffic object of a first category in each frame image of the plurality of frame images and a template image of each second category, each second category being a subcategory of the first category; determining a subcategory having the highest confidence level of the first category of traffic objects in the image of each frame based on the second similarity level and a confidence level of the subcategory; The image processing method according to claim 5.

8. Determining correction information for a subcategory of the traffic object of the first category, the maximum reliability of which does not satisfy a preset condition, based on a comparison result of the reliability of the subcategory of the traffic object of the first category in the plurality of frame images, the maximum reliability of which is the highest; in response to a confidence level of a subdivision category having a highest confidence level for a first category traffic object in a first image of the plurality of frame images satisfying a third preset condition and a confidence level of a subdivision category having a highest confidence level for a first category traffic object in a second image of the plurality of frame images not satisfying the third preset condition, setting the subdivision category having the highest confidence level for the first category traffic object in the second image to the same category as the subdivision category having the highest confidence level for the first category traffic object in the first image. The image processing method according to claim 6.

9. and further comprising: outputting hint information indicating that a classification result of the traffic object cannot be determined in response to any of the reliability of the subcategory having the maximum reliability of the first category of the traffic object in the plurality of frame images not satisfying a third preset condition. The image processing method according to claim 8.

10. An image processing apparatus, comprising: an acquisition unit, a first determination unit, a second determination unit and a third determination unit; The acquisition unit is configured to acquire a video stream collected by an image acquisition device installed on a traveling machine; The first determining unit is configured to determine a plurality of frame images including a particular traffic object from the video stream; The second determination unit is configured to determine a confidence level of a category of a particular traffic object in an image of each frame of the plurality of frame images; The image processing device, wherein the third determination unit is configured to determine correction information for a category whose reliability does not satisfy a preset condition based on a comparison result of the reliability of the category of a specific traffic object in the plurality of frame images.

11. A computer readable storage medium storing a computer program which, when executed by a processor, causes the processor to implement the steps of the image processing method according to any one of claims 1 to 9.

12. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, the electronic device realizing the steps of the image processing method according to any one of claims 1 to 9 when the processor executes the program.

13. A computer program product comprising a non-transitory computer-readable storage medium having a computer program stored thereon, the computer program product implementing the steps of the image processing method according to any one of claims 1 to 9 when the computer program is read and executed by a computer.

Images