An image processing method and apparatus

Abstract

Embodiments of the present application disclose an image processing method and device, the method comprising: acquiring region information and category information of at least two target objects in a first image; grouping the at least two target objects in the first image according to the region information and the category information to obtain group information of at least one group; adjusting region information of the target objects in the group according to the group information and an intra-group constraint relationship; and identifying the target objects in the group based on the adjusted region information. The embodiments of the present application can improve the accuracy of identifying target objects in an image.

Description

Technical Field

[0001] This application relates to the field of autonomous driving, and more particularly to an image processing method and apparatus. Background Technology

[0002] In recent years, with the continuous development of automotive intelligence, traditional automotive functions can no longer meet people's needs. More intelligent automotive accessories can bring people a modern experience of interconnectedness. Traffic sign and signal light detection and recognition is one of the main application scenarios of vehicle-mounted imaging systems. The detection and marking (or labeling) of traffic lights can, to a certain extent, improve the safety performance of vehicles during operation, enhance the intelligence of vehicle systems, and thus improve the user experience of using vehicle systems.

[0003] Unlike other general object detection scenarios (such as pedestrian detection), object detection and labeling in traffic light scenarios have unique characteristics. For example, traffic lights in the same scene (on the same light frame) often have the same size and follow a predetermined arrangement rule (such as multiple red or green lights arranged in a straight line). Given these unique characteristics, using object detection methods applicable to a wide range of scenarios for traffic light detection and labeling cannot meet the constraints on the size and arrangement of the bounding boxes. If the detected traffic lights are of inconsistent sizes and irregularly arranged, the labeling results often present a cluttered visual effect to the user, making them feel that the vehicle's infotainment system's detection is inaccurate. Furthermore, in the process of labeling massive datasets, manual labeling and adjustments often introduce errors. Summary of the Invention

[0004] This application provides an image processing method and apparatus that improves the accuracy of identifying target objects in images.

[0005] In a first aspect, embodiments of this application provide an image processing method, which is applied to a mobile device, or to a chip or circuit configured in a mobile device or server, comprising:

[0006] Obtain region information and category information of at least two target objects in the first image; group the at least two target objects in the first image according to the region information and the category information to obtain group information of at least one group; adjust the region information of the target objects in the group according to the group information and the constraints within the group; identify the target objects in the group based on the adjusted region information.

[0007] Based on the category and region information of the target object in the first image, group information is determined. The region information within each group is adjusted based on the group information and intra-group constraints. The target object is then identified based on the adjusted region information. This enables traffic light detection and identification in special scenarios, providing users with more accurate detection results and more orderly identification results, improving the user's visual experience and overall user experience.

[0008] In one possible design, the intragroup constraints include at least one of size consistency constraints and position consistency constraints.

[0009] In another possible design, when the intra-group constraint relationship is the size consistency constraint, the original size of each target object in the group is determined based on the region information of each target object in the group; the scaling factor of each target object in the group is determined based on the original size; and the region information of each target object in the group is updated based on the scaling factor.

[0010] Based on size consistency constraints and group information, the region information of the target object in the first image is fine-tuned to determine more accurate region information, thereby improving the accuracy of identifying the target object.

[0011] In another possible design, the original size includes the original vertical size and the original horizontal size, and the scaling factor includes a horizontal scaling factor and a vertical scaling factor; based on the original horizontal size, the target horizontal size of each target object in the group is determined; based on the original vertical size, the target vertical size of each target object in the group is determined; the target horizontal size of each target object in the group is divided by the original horizontal size to obtain the horizontal scaling factor; and the target vertical size of each target object in the group is divided by the original vertical size to obtain the vertical scaling factor.

[0012] In another possible design, when the intra-group constraint relationship is a positional consistency constraint, the original reference point position of each target object in the group is determined based on the region information of each target object in the group; the target reference point position of each target object in the group is determined based on the original reference point position of each target object in the group; and the region information of each target object in the group is updated based on the original reference point position and the target reference point position of each target object in the group.

[0013] Based on positional consistency constraints and group information, the regional information of the target object in the first image is fine-tuned to determine more accurate regional information, thereby improving the accuracy of identifying the target object.

[0014] In another possible design, linear regression is performed on the original reference point positions of the target objects within the group to obtain a linear regression equation; based on the linear regression equation, the target reference point position of each target object within the group is determined.

[0015] In another possible design, based on the region information and the category information, at least two target objects in the first image are clustered to obtain at least one group; the intra-class variance of each group is determined; when the intra-class variance of all at least one group is less than a preset threshold, the target objects contained in each group are used as the group information of the at least one group. Grouping by calculating intra-class variance improves the accuracy of the group information.

[0016] In another possible design, the target object within the group is identified based on the category information and the adjusted region information.

[0017] Secondly, embodiments of this application provide an image processing apparatus, including:

[0018] The acquisition module is used to acquire region information and category information of at least two target objects in the first image;

[0019] The processing module is configured to group at least two target objects in the first image according to the region information and the category information to obtain group information of at least one group; adjust the region information of the target objects in the group according to the group information and the intra-group constraints; and identify the target objects in each group based on the adjusted region information.

[0020] In one possible design, the intragroup constraints include at least one of size consistency constraints and position consistency constraints.

[0021] In another possible design, the processing module is further configured to determine the original size of each target object in the group based on the region information of each target object in the group when the constraint relationship within the group is a size consistency constraint;

[0022] Based on the original size, determine the scaling factor for each target object within the group;

[0023] The region information of each target object within the group is updated based on the scaling factor.

[0024] Wherein, the original size includes the original vertical size and the original horizontal size, and the scaling factor includes the horizontal scaling factor and the vertical scaling factor;

[0025] In another possible design, the processing module is also used to determine the target lateral dimension of each target object within the group based on the original lateral dimension.

[0026] Based on the original longitudinal dimensions, determine the target longitudinal dimensions of each target object within the group;

[0027] The horizontal scaling factor is obtained by dividing the target horizontal dimension of each target object in the group by the original horizontal dimension; and the vertical scaling factor is obtained by dividing the target vertical dimension of each target object in the group by the original vertical dimension.

[0028] In another possible design, the processing module is further configured to determine the original reference point position of each target object in the group based on the region information of each target object in the group when the intra-group constraint relationship is a position consistency constraint;

[0029] Based on the original reference point position of each target object in the group, determine the target reference point position of each target object in the group;

[0030] Based on the original reference point position and the target reference point position of each target object in the group, the regional information of each target object in the group is updated.

[0031] In another possible design, the processing module is also used to perform linear regression on the original reference point position of the target object within the group to obtain a linear regression equation;

[0032] Based on the linear regression equation, the target reference point position of each target object within the group is determined.

[0033] In another possible design, the processing module is further configured to cluster the at least two target objects in the first image to obtain at least one group based on the region information and the category information; determine the intra-class variance of each group in the at least one group; and when the intra-class variance of the at least one group is less than a preset threshold, use the target objects contained in each group as the group information of the at least one group.

[0034] In another possible design, the processing module is further configured to identify the target object within the group based on the category information and the adjusted region information.

[0035] The operation and beneficial effects performed by the image processing device can be found in the method and beneficial effects described in the first aspect above, and will not be repeated here.

[0036] Thirdly, this application provides an image processing apparatus, the image processing apparatus including a processor and a memory, the memory being used to store a computer program; the processor being used to execute the computer program stored in the memory to cause the image processing apparatus to perform the method as described in any one of the first aspects.

[0037] Fourthly, this application provides an image processing apparatus, which can be a mobile device, a device in a vehicle-mounted imaging system or a server, or a device compatible with a vehicle-mounted imaging system. The image processing apparatus can also be a chip system. The image processing apparatus can execute the method described in the first aspect. The functions of the image processing apparatus can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. These modules can be software and / or hardware. The operations performed by the image processing apparatus and its beneficial effects are described in the first aspect above, and will not be repeated here.

[0038] Fifthly, this application provides an image processing apparatus comprising a processor and a memory, the memory for storing a computer program, the processor for calling the computer program stored in the memory, and the processor for performing the method as described in any one of the first aspects.

[0039] In a sixth aspect, this application provides a computer-readable storage medium for storing a computer program that, when executed, causes the method described in any one of the first aspects to be implemented.

[0040] In a seventh aspect, this application provides a computer program product including a computer program that, when executed, causes the method described in any one of the first aspects to be implemented. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of this application or the background art, the accompanying drawings used in the embodiments of this application or the background art will be described below.

[0042] Figure 1 This is a schematic diagram of the overall framework of a vehicle-mounted imaging system provided in an embodiment of this application;

[0043] Figure 2 This is a schematic flowchart of an image processing method provided in an embodiment of this application;

[0044] Figure 3 This is a schematic diagram illustrating a group information acquisition process provided in an embodiment of this application;

[0045] Figure 4 This is a schematic diagram of a process for adjusting regional information provided in an embodiment of this application;

[0046] Figure 5 This is a schematic diagram of a network model training process provided in an embodiment of this application;

[0047] Figure 6 This is a schematic diagram of a network model provided in an embodiment of this application;

[0048] Figure 7 This is a schematic diagram of the structure of an image processing device provided in an embodiment of this application;

[0049] Figure 8 This is a schematic diagram of another image processing device provided in an embodiment of this application. Detailed Implementation

[0050] The embodiments of this application are described below with reference to the accompanying drawings.

[0051] like Figure 1 As shown, Figure 1 This is a schematic diagram of the overall framework of a vehicle-mounted imaging system provided in an embodiment of this application. The imaging system includes an application layer (APP), a framework layer, and a hardware abstraction layer (HAL). The framework layer includes a traffic light detection and labeling module. This module acquires input data from the current image, detects traffic light targets, and labels them. Traffic light target detection relies on a constraint-based labeling dataset and a constraint-trained network model. In the traffic light labeling process, constraints between traffic lights are used to fine-tune the region information (anchors) of the detected traffic lights in the image before labeling.

[0052] like Figure 2 As shown, Figure 2 This is a flowchart illustrating an image processing method provided in an embodiment of this application. The method mainly includes the following steps:

[0053] S201, Obtain region information and category information of at least two target objects in the first image.

[0054] The first image can be a video image captured by the vehicle's imaging system, and can be a single frame from the video image. The target object can be a traffic light in the first image, which can be identified by a bounding box in the first image. The bounding box can be square, circular, or any other shape. The region information is the position of the bounding box of the target object in the first image. For example, if the bounding box is square, the region information is the coordinates of each corner point of the bounding box in the first image. If the bounding box is circular, the region information is the coordinates of multiple points on the circumference of the bounding box. The category information can be of types such as straight-ahead red light target, straight-ahead green light target, turn signal red light target, turn signal green light target, stopwatch red light target, or stopwatch green light target.

[0055] Optionally, the first image can be preprocessed. Preprocessing includes, but is not limited to: low-light enhancement, high-dynamic range (HDR) enhancement, rain removal, defogging, de-reflection, normalization, etc.

[0056] S202, based on the region information and the category information, group at least two target objects in the first image to obtain group information of at least one group.

[0057] Specifically, based on the region information and the category information, at least two target objects in the first image can be clustered to obtain at least one group; the intra-class variance of each group in the at least one group can be determined; when the intra-class variance of all at least one group is less than a preset threshold, the target objects contained in each group are used as the group information of the at least one group. When the intra-class variance of any one of the at least one groups is not less than the preset threshold, the number of groups is increased, multiple target objects in the first image are clustered to obtain new groups, and then the intra-class variance is recalculated to determine the group information.

[0058] For example, such as Figure 3 As shown, Figure 3 This is a schematic diagram illustrating a group information acquisition process provided in an embodiment of this application. It includes:

[0059] The first step is to obtain the region information (anchors) and category information (class) of the target object.

[0060] Os = {O1, O2, ..., O n}

[0061] Where Os is the set of target objects (traffic lights) detected in the first image. Os includes target object O1, target object O2, ..., target object O n .

[0062] anchors={anchor1,anchor2,……,anchor n}

[0063] Here, `anchors` is a list of region information for multiple target objects in the first image. `anchors` includes the region information of target object O1 (anchors1), target object O2 (anchors2), ..., target object O... n regional information anchors n .

[0064] class={class1,class2,……,class n}

[0065] Here, `class` is a list of category information for multiple target objects in the first image. `class` includes the category information `class1` for target object O1, `class2` for target object O2, ..., and so on. n Category information n .

[0066] The second step is to initialize the number of groups K = 2;

[0067] The third step is to use the K-means algorithm to cluster the target objects in Os to obtain K groups;

[0068] The fourth step is to calculate the within-class variance of each of the K groups:

[0069] Gvars = {var1, ..., var...} K}

[0070] Where Gvars is the within-class variance of the K groups, var1 is the within-class variance of the first group, ..., var K Let be the within-class variance of the Kth group.

[0071] The fifth step is to determine whether the intra-class variance of each of the K groups is less than the preset threshold TH_var.

[0072] Step 6: If the intra-class variance of all K classes is less than the preset threshold TH_var, then the current clustering result is output as group information.

[0073] Step 7: If at least one group has an intraclass variance greater than the preset threshold TH_var, then update the number of groups K.

[0074] K = K + 1

[0075] Repeat the above steps (steps 3 to 7).

[0076] S203, adjust the region information of the target object within the group according to the group information and the constraints within the group.

[0077] The intra-group constraints can include at least one of size consistency constraints and position consistency constraints. The region information of target objects within one or more groups can be adjusted based on size consistency constraints or position consistency constraints. Alternatively, the region information of target objects within one or more groups can be adjusted based on both size consistency constraints and position consistency constraints.

[0078] like Figure 4 As shown, Figure 4 This is a schematic flowchart illustrating a region information adjustment process provided in an embodiment of this application. It includes the following steps: obtaining group information for at least one group; when the constraint relationship within the group is a size consistency constraint, determining the original size and target size of the target object; calculating a scaling factor based on the original size and target size; and scaling the annotation box of the target object based on the scaling factor. When the constraint relationship within the group is a position consistency constraint, determining a linear regression equation based on the original reference point position of the target object; calculating the target reference point position through the linear regression equation; and translating the annotation box of the target object based on the original reference point position and the target reference point position. The reference point can be the center point or a point at other locations. The following details the process of adjusting the region information:

[0079] In one implementation, when the intra-group constraint is a size consistency constraint, the original size of each target object in the group is determined based on the region information of each target object in the group; a scaling factor is determined based on the original size of each target object in the group; and the region information of each target object in the group is updated based on the scaling factor. For example, the bounding box of the target object is scaled.

[0080] The original size of the target object can be the size of the target object's annotation box. The original size of the target object includes the original vertical dimension and the original horizontal dimension. The original vertical dimension can be the height of the target object's annotation box, and the original horizontal dimension can be the width of the target object's annotation box. The target size of the target object includes the target vertical dimension and the target horizontal dimension. The target vertical dimension can be the height of the target object's annotation box, and the target horizontal dimension can be the width of the target object's annotation box.

[0081] Furthermore, the scaling factors include a horizontal scaling factor and a vertical scaling factor. The target horizontal size of each target object within the group can be determined based on the original horizontal size; for example, the average or weighted average of the original horizontal sizes of multiple target objects within the group can be calculated as the target horizontal size. Similarly, the target vertical size of each target object within the group can be determined based on the original vertical size; for example, the average or weighted average of the original vertical sizes of multiple target objects within the group can be calculated as the target vertical size. Then, the target horizontal size of each target object within the group is divided by the original horizontal size to obtain the horizontal scaling factor; and the target vertical size of each target object within the group is divided by the original vertical size to obtain the vertical scaling factor.

[0082] For example, a group G = {Os, E, Re} can be composed of the set of target objects (traffic lights) Os in the first image, the adjacency list E, and the intra-group constraints Re. Re includes size consistency constraints and position consistency constraints.

[0083] Os = {O1, O2, ..., O n}

[0084] Where Os is the set of target objects (traffic lights) detected in the first image. Os includes target object O1, target object O2, ..., target object O n .

[0085] E={e 1,1 ,e 1,2 ,…e i,j ,…e j,i ,e n,n}

[0086] Among them, the adjacency list E can be accessed through e 1,1 ,e 1,2 ,…e i,j ,…e j,i ,e n,n It indicates. e 1,2 Indicates whether target object O1 and target object O2 are in the same group, ..., e i,j Represents target object O i With target object O j Whether or not they are in a group.

[0087] When the target O i With target O j When in the same group:

[0088] e i,j =e j,i =1

[0089] When the target O i With target O j When not in the same group:

[0090] e i,j =e j,i =0

[0091] Meanwhile, when i = j:

[0092] e i,j =0

[0093] The following details how to adjust the region information of the target object within each group under the constraint that Re is a size consistency relationship:

[0094] For a group G = {Os, E, Re}, we can obtain a list of target objects Os′ and a corresponding list of anchors within the group G based on the adjacency matrix E:

[0095] Os′={O i O j ...}

[0096] Among them, group Os′ includes target object O i Target object O j wait.

[0097] anchors′={anchor i anchor j ...}

[0098] The anchor list includes target objects O within group Os′. i regional information anchor i Target object O j regional information anchor j etc.

[0099] When the intra-group constraint Re is a size consistency relationship:

[0100] Calculate the original vertical dimension (height) and original horizontal dimension (width) of each target object in the anchor list:

[0101] sizes′={(w i ,h i ),(w j ,h j )……}

[0102] Here, sizes′ includes the target object O within group Os′. i The width and height (w) of the annotation boxi ,h i ), target object O j The width and height (w) of the annotation box j ,h j )wait.

[0103] Calculate the target vertical dimension and target horizontal dimension of each target object within group Os′. The calculation method includes, but is not limited to, the mean method, the weighted average method, etc.

[0104] dstsize = (dst_w, dst_h)

[0105] Where dstsize is the target size, dst_w is the target horizontal size, and dst_h is the target vertical size.

[0106] Calculate the horizontal scaling factor of the target object:

[0107] scales_w′={scale_w i ,scale_w j ...}

[0108] Here, scales_w′ includes the target object O i Horizontal scaling factor scale_w i Target object O j Horizontal scaling factor scale_w j wait.

[0109] scale_w k =dst_w / w k

[0110] Among them, scale_w k This represents the horizontal scaling factor of the k-th target object in scales_w′. dst_w is the horizontal dimension of the target object, and w... k This represents the original horizontal dimension of the k-th target object.

[0111] Calculate the vertical scaling factor of the target object:

[0112] scales_h′={scale_h i ,scale_h j ...}

[0113] Here, scales_h′ includes the target object O i Vertical scaling factor scale_h i Target object O j Vertical scaling factor scale_h j wait.

[0114] scale_h k =dst_h / h k

[0115] Among them, scale_h k This represents the vertical scaling factor of the k-th target object in scales_h′. dst_w is the vertical dimension of the target object, where w is the vertical scaling factor. k This represents the original vertical dimension of the k-th target object.

[0116] Finally, the annotation boxes of the target objects in group Os′ are scaled according to the vertical and horizontal scaling factors of the target objects in group Os′.

[0117] In another implementation, when the intra-group constraint is a positional consistency constraint, the original reference point position of each target object in the group is determined based on the region information of each target object within the group; the target reference point position of each target object in the group is determined based on the original reference point position of each target object within the group; and the region information of each target object in the group is updated based on the original reference point position and the target reference point position. For example, the bounding box of the target object can be translated. The reference point can be the center point or a point at another location.

[0118] The original reference point position of the target object can be the position of the reference point of the target object's annotation box. The target reference point position of the target object is the position of the reference point after the target object's annotation box has been translated.

[0119] Furthermore, linear regression can be performed on the original reference point positions of the target objects within the group to obtain a linear regression equation; based on the linear regression equation, the target reference point position of each target object within the group can be determined.

[0120] The following details how to adjust the region information of the target object within each group under the constraint that the intra-group constraint Re is a positional consistency constraint:

[0121] Calculate the original reference point position of each target object within a group in the anchor list:

[0122] opoints′={point i ,point j ...}

[0123] The reference point list opoints′ includes a target object O within a group Os′. i The original reference point position of the annotation box. i Target object Oj The original reference point position of the annotation box. j wait.

[0124] A first-order linear regression was performed on the positions of all original reference points in the reference point list opoints′, and the linear equation after regression was obtained:

[0125] y = ax + b

[0126] Where a and b are parameters, x is the coordinate in the x-axis direction, and y is the coordinate in the y-axis direction.

[0127] Calculate the reference points for the bounding boxes of each target object in the anchor list, including the following optional methods:

[0128] The first optional method is to keep the original reference point's coordinates along the x-axis of the first image unchanged, and calculate the projection of the original reference point's position onto the regression equation as the target reference point position:

[0129] point i ′=(x i ,y i ′)

[0130] y i ′=a*x i +b

[0131] Where, point i The target reference point position includes the coordinates x along the x-axis. i and the coordinates in the y-axis direction i ′.

[0132] The second option is to retain the original reference point's coordinates along the y-axis of the first image and calculate the projection of the original reference point onto the regression equation, which will then be used as the target reference point's position.

[0133] point i ′=(x i ′,y i )

[0134] y i =a*x i ′+b

[0135] Where, point i The target reference point position includes the coordinates x along the x-axis. i The coordinates of the y-axis and the y-axis. i .

[0136] The third option is to calculate the vertical projection of the original reference point position of each target object within the group onto the regression equation, and use this as the target reference point position:

[0137] point i ′=(x i ′,y i ′)

[0138] y i ′=a*x i ′+b

[0139] Where, point i The target reference point position includes the coordinates x along the x-axis. i The coordinates of the y-axis and the y-axis. i ′.

[0140] Finally, based on the original reference point positions of each target object within group Os′... i relative to the target reference point i The `translate` operator moves the bounding boxes of each target object within group `Os`. For example, it moves the reference point of the target object's bounding box from its original reference point position `point`. i Move to the target reference point i ′.

[0141] S204, Based on the adjusted area information, identify the target object within the group.

[0142] Optionally, the target object within the group can be identified based on category information and the adjusted region information.

[0143] Optionally, a first image can be input into a trained network model to obtain region information and category information of at least two target objects in the first image; based on the region information and category information, the at least two target objects in the first image are grouped to obtain group information of at least one group; based on the group information and intra-group constraints, the region information of the target objects within the group is adjusted; based on the adjusted region information, the target objects within the group are identified; and the first image with the identified target objects is output. The trained network model can be deployed on a mobile device, a vehicle imaging system, or a server. After preprocessing the first image, it can be input into the trained network model.

[0144] By optimizing the loss function during model training and incorporating a penalty factor based on group information and inter-group constraints, the regressed region information is made to conform to the constraints to a certain extent. The network model is then trained using these parameters, thereby improving the accuracy of its predictions.

[0145] In this embodiment, group information is determined based on the category and region information of the target object in the first image. The region information within the group is adjusted according to the group information and intra-group constraints. The target object is then identified based on the adjusted region information. This enables traffic light detection and identification in special scenarios, providing users with more accurate detection results and more orderly identification results, thereby improving the user's visual experience and overall user experience.

[0146] like Figure 5 As shown, Figure 5 This is a schematic diagram illustrating a network model training process provided in an embodiment of this application. The method mainly includes the following steps:

[0147] S501, Obtain the dataset, which includes multiple frames of images to be trained.

[0148] Specifically, based on actual needs, the data collection requirements for traffic light (target object) detection can be determined, and data collection and cleaning can be performed accordingly. The specific process is as follows:

[0149] The actual needs include, but are not limited to:

[0150] Traffic light category information; output a traffic light category information table:

[0151] ss_class={class1,class2,...,class n}

[0152] Where n is the number of traffic light categories; the traffic light categories include, but are not limited to: straight-ahead red light target, straight-ahead green light target, turn signal red light target, turn signal green light target, stopwatch red light target, stopwatch green light target, etc.

[0153] The scenarios in which traffic lights exist should be output as a traffic light scenario information table:

[0154] ss_scene={scene1,scene2,……,scene m}

[0155] Where m represents the number of traffic light scenarios; the traffic light scenarios include, but are not limited to: time scenarios, vehicle type scenarios, location scenarios, etc.

[0156] Based on actual needs, video images of traffic lights are acquired through a vehicle-mounted imaging system to obtain a dataset, which is then cleaned (e.g., low-light enhancement, HDR enhancement, rain removal, fog removal, glare removal, normalization, etc.). The dataset includes traffic light category information and traffic light scene information.

[0157] S502, respectively identify the region information, category information, group information and intra-group constraint relationship of the target object in each frame of the multi-frame image.

[0158] In the first stage, a pre-trained object detection network and object detection labeling tool are used to label the target objects (traffic lights) in the dataset, and labeling files for traffic lights that meet preset accuracy requirements are saved. The labeling file includes at least the anchor information and class information of the traffic light. During this process, the anchor information and class information of the traffic lights can be repeatedly optimized, and the final labeling file is saved, thus ensuring the accuracy and consistency of the labeling files.

[0159] The second stage involves identifying the group information of target objects in each frame of a multi-frame image dataset based on clustering algorithms. This mainly includes the following steps:

[0160] First, the region information and category information of the target object (traffic light) are obtained from the identification file. Based on the category information and region information of the target object, the target objects in each frame of the image are clustered (for example, multiple traffic lights on the same traffic light frame are marked as a group) to obtain group information. The group information is manually optimized, and the optimized group information is identified for the target object.

[0161] The identified group information can be represented using an adjacency matrix:

[0162] Os = {O1, O2, ..., O n}

[0163] Where Os is the set of target objects (traffic lights) detected in the current frame image, and n is the number of target objects (traffic lights) detected.

[0164] For a labeled group G, it consists of a target object set Os and an adjacency list E:

[0165] G = {Os, E}

[0166] E={e 1,1 ,e 1,2 ,……,e n,n}

[0167] When the target object O i With target object O j When in the same group:

[0168] e i,j =e j,i =1

[0169] When the target object O i With target object O j When not in the same group:

[0170] e i,j =e j,i =0

[0171] Meanwhile, when i = j:

[0172] e i,j =0

[0173] Secondly, intra-group constraints are identified for the target objects of the identified group information. These constraints include, but are not limited to, size consistency constraints, position consistency constraints, etc. Therefore, the identified group G can be composed of the target set Os, the adjacency list E, and the intra-group constraint Re.

[0174] G = {Os, E, Re}

[0175] S503, based on the group information and the intra-group constraints, adjust the region information of the target objects within the same group.

[0176] Specifically, for target objects within the same group, the regional information of each target object in each group is fine-tuned based on the group's constraints. Then, the identifier file is updated and saved. This identifier file includes, but is not limited to, the following identifier information: fine-tuned regional information, category information, group information, and group constraint information.

[0177] S504, based on the adjusted region information, the identified category information, the identified group information, and the identified intra-group constraint relationships, the network model to be trained is trained to obtain the trained network model.

[0178] Specifically, based on the adjusted region information of each frame, each frame can be re-labeled to obtain a labeled image. Then, multiple frames of images and the labeled images are input into the input layer of the network model for training to obtain a trained network model.

[0179] Figure 6 This is a schematic diagram of a network model provided in an embodiment of this application. Multiple frames of images are input to the input layer of the network model. Stacked convolutional neural networks are used for feature extraction of each frame, such as extracting the region information, category information, group information, and intra-group constraints of traffic lights contained in each frame. The output layer of the network model can obtain the following information: the region (anchor) information of the traffic lights, the category (class) information of the traffic lights, the group information of the traffic lights, and the intra-group constraints.

[0180] The loss function of the network model is as follows:

[0181] loss=α*loss(class)+β*loss(anchor)+γ*loss(G)

[0182] Wherein, α, β, and γ are weighting factors (also called weighting factors or weight coefficients), and α+β+γ=1; the loss(class) is the category information loss of the target object; the loss(anchor) is the region information loss of the target object; and the loss(G) is the group information loss of the target object.

[0183] Furthermore, the loss(G) = ω1*loss(E) + ω2*oss(Re) + ω3*loss(Re&anchors)

[0184] Wherein, ω1, ω2, and ω3 are weighting coefficients (also called weighting factors or weight coefficients), and ω1 + ω2 + ω3 = 1; the loss(E) is the group adjacency loss, the loss(Re) is the intra-group constraint relationship loss, and the loss(Re & anchors) is the intra-group constraint relationship penalty factor.

[0185] Finally, based on the target objects identified in each frame of the above dataset and the loss function of the constructed model to be trained, the parameters of the network model to be trained are trained and tuned, and the model structure and parameters that meet the preset conditions are used as the model structure and parameters of the trained network model.

[0186] In this embodiment, the target object's region information, category information, group information, and intra-group constraints are added to the output layer of the network model to be trained. The loss function for model training is optimized, and a penalty factor is added based on the group information and inter-group constraints to ensure that the regressed region information conforms to the constraints to a certain extent. Finally, the network model to be trained is trained based on the target object's region information, category information, group information, and intra-group constraints in each frame of the image, along with the constructed loss function. This improves the accuracy of the network model's predictions.

[0187] It should be noted that the steps in the above embodiments are not in any particular order, and the above embodiments can be combined or separated, which is not limited in this application.

[0188] This application embodiment can divide a mobile device into functional modules based on the above method example. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into one processing module. The integrated modules can be implemented in hardware or software functional modules. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation. The following description uses the division of functional modules according to each function as an example.

[0189] The above, combined with Figures 2-5 The methods provided in the embodiments of this application are described in detail below. Figure 7 This application provides a detailed description of the image processing apparatus provided in its embodiments. It should be understood that the descriptions of the apparatus embodiments correspond to the descriptions of the method embodiments; therefore, any details not described in detail can be found in the above method embodiments, and for the sake of brevity, will not be repeated here.

[0190] Please see Figure 7 , Figure 7 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of this application. The image processing apparatus may include an acquisition module 701 and a processing module 702. In one possible design, the image processing apparatus can implement the steps or processes executed by the mobile device corresponding to the method embodiments described above. For example, it can be a mobile device, or a chip or circuit configured in a mobile device or server.

[0191] The acquisition module 701 is used to acquire region information and category information of at least two target objects in the first image;

[0192] The processing module 702 is configured to group at least two target objects in the first image according to the region information and the category information to obtain group information of at least one group; adjust the region information of the target objects in the group according to the group information and the intra-group constraints; and identify the target objects in each group based on the adjusted region information.

[0193] The intra-group constraint relationship includes at least one of size consistency constraint and position consistency constraint.

[0194] Optionally, the processing module 702 is further configured to, when the intra-group constraint relationship is a size consistency constraint,

[0195] Based on the region information of each target object within the group, determine the original size of each target object within the group; based on the original size, determine the scaling factor of each target object within the group.

[0196] The region information of each target object within the group is scaled according to the scaling factor.

[0197] Wherein, the original size includes the original vertical size and the original horizontal size, and the scaling factor includes the horizontal scaling factor and the vertical scaling factor;

[0198] Optionally, the processing module 702 is further configured to determine the target horizontal dimension of each target object in the group based on the original horizontal dimension; and to determine the target vertical dimension of each target object in the group based on the original vertical dimension.

[0199] The horizontal scaling factor is obtained by dividing the target horizontal dimension of each target object in the group by the original horizontal dimension; and the vertical scaling factor is obtained by dividing the target vertical dimension of each target object in the group by the original vertical dimension.

[0200] Optionally, the processing module 702 is further configured to determine the original reference point position of each target object in the group based on the region information of each target object in the group when the intra-group constraint relationship is a position consistency constraint;

[0201] Based on the original reference point position of each target object in the group, determine the target reference point position of each target object in the group;

[0202] Based on the original reference point position and the target reference point position of each target object in the group, the regional information of each target object in the group is translated.

[0203] Optionally, the processing module 702 is further configured to perform linear regression on the original reference point position of the target object within the group to obtain a linear regression equation;

[0204] Based on the linear regression equation, the target reference point position of each target object within the group is determined.

[0205] Optionally, the processing module 702 is further configured to cluster the at least two target objects in the first image to obtain at least one group based on the region information and the category information;

[0206] Determine the within-class variance of each group in the at least one group;

[0207] When the intra-class variance of the at least one group is less than a preset threshold, the target objects contained in each group are used as the group information of the at least one group.

[0208] Optionally, the processing module 702 is further configured to identify the target object within the group based on the category information and the adjusted region information.

[0209] It should be noted that the implementation of each module can also be referenced accordingly. Figures 2-5 The corresponding description of the method embodiments shown above describes the methods and functions performed by the mobile device in the above embodiments.

[0210] Figure 8 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of this application. This image processing apparatus can be applied to, for example... Figure 1 In the system shown, the functions of the mobile device in the above method embodiments are executed, or the steps or processes executed by the mobile device in the above method embodiments are implemented. The image processing device can be a vehicle, or an in-vehicle system configured in a vehicle, or configured in a server.

[0211] like Figure 8 As shown, the image processing apparatus includes a processor 801 (e.g., a graphics processing unit (GPU) or a central processing unit (CPU)) and a transceiver 802. Optionally, the image processing apparatus also includes a memory 803. Optionally, the image processing apparatus may also include a camera to capture video images (multi-frame images) during movement.

[0212] The processor 801, transceiver 802, and memory 803 can communicate with each other via internal connections to transmit control and / or data signals. The memory 803 stores computer programs, and the processor 801 retrieves and runs these programs from the memory 803 to control the transceiver 802 to transmit and receive signals. Optionally, the image processing device may also include an antenna for transmitting uplink data or uplink control signaling output by the transceiver 802 via wireless signals.

[0213] The processor 801 and memory 803 can be combined into a single processing device. The processor 801 executes the program code stored in the memory 803 to achieve the aforementioned functions. In specific implementations, the memory 803 can be integrated into the processor 801 or independent of it. The processor 801 can be combined with... Figure 7 The corresponding processing module in [the system / processing module].

[0214] The transceiver 802 described above can also be referred to as a transceiver unit or transceiver module. The transceiver 802 may include a receiver (or receiver circuit) and a transmitter (or transmitter circuit). The receiver is used to receive signals, and the transmitter is used to transmit signals.

[0215] It should be understood that Figure 8 The image processing device shown can achieve Figures 2-5 The methods illustrated in the embodiments involve various processes of a mobile device. The operations and / or functions of each module in the image processing device are respectively for implementing the corresponding processes in the above method embodiments. For details, please refer to the descriptions in the above method embodiments; to avoid repetition, detailed descriptions are appropriately omitted here.

[0216] The processor 801 described above can be used to perform the actions implemented internally by the mobile device as described in the preceding method embodiments, while the transceiver 802 can be used to perform the actions described in the preceding method embodiments of sending data to or receiving data from other devices by the mobile device. Please refer to the descriptions in the preceding method embodiments for details, which will not be repeated here.

[0217] The processor 801 can be a central processing unit, a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor 801 can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a digital signal processor and a microprocessor, etc. The communication bus 804 can be a peripheral component interconnect standard PCI bus or an extended industry standard structure EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8The bus is represented by a single thick line, but this does not imply that there is only one bus or one type of bus. The communication bus 804 is used to implement communication between these components. In this embodiment, the transceiver 802 is used for signaling or data communication with other node devices. The memory 803 may include volatile memory, such as nonvolatile random access memory (NVRAM), phase change RAM (PRAM), magnetoresistive RAM (MRAM), etc., and may also include non-volatile memory, such as at least one disk storage device, electrically erasable programmable read-only memory (EEPROM), flash memory devices, such as NOR flash memory or NAND flash memory, and semiconductor devices, such as solid-state disks (SSDs). Optionally, the memory 803 may also be at least one storage device located remotely from the aforementioned processor 801. Optionally, the memory 803 may also store a set of computer program code or configuration information. Optionally, the processor 801 can also execute programs stored in the memory 803. The processor can cooperate with the memory and transceiver to execute any of the methods and functions of the mobile device in the above-described embodiments.

[0218] This application also provides a chip system including a processor for supporting mobile devices to implement the functions involved in any of the above embodiments, such as generating or processing the adjustment of region information of target objects within each group based on group information and intra-group constraints, as involved in the above methods. In one possible design, the chip system may further include a memory for storing necessary program instructions and data for the mobile device. The chip system may be composed of chips or may include chips and other discrete devices. The inputs and outputs of the chip system correspond to the receiving and transmitting operations of the mobile device in the method embodiments, respectively.

[0219] According to the method provided in the embodiments of this application, this application also provides a computer program product, which includes: a computer program that, when run on a computer, causes the computer to perform... Figures 2-5 The method of any one of the embodiments shown.

[0220] According to the method provided in the embodiments of this application, this application also provides a computer-readable medium storing a computer program, which, when run on a computer, causes the computer to perform... Figures 2-5 The method of any one of the embodiments shown.

[0221] In this application, the term "multiple" in the embodiments refers to two or more objects; "and / or" describes the relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone, where A and B can be singular or plural; the character " / " generally indicates that the preceding and following associated objects are in an "or" relationship. Furthermore, unless otherwise stated, the term "first" in the embodiments of this application is only used for name identification and is not used to limit the order, sequence, priority, or importance of multiple objects. This rule also applies to "second," "third," and "fourth," etc.

[0222] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., high-density digital video discs (DVDs)), or semiconductor media (e.g., solid-state disks (SSDs)).

[0223] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image processing method, characterized in that, The method includes: Obtain region and category information for at least two target objects in the first image; Based on the region information and the category information, the at least two target objects are grouped to obtain group information for at least one group; Based on the group information and intra-group constraints, the region information of the target object within the group is adjusted. The group information includes the target object contained in each group of the at least one group, and the intra-group constraints include at least one of size consistency constraints and position consistency constraints. Based on the adjusted region information, the target object within the group is identified.

2. The method as described in claim 1, characterized in that, The step of adjusting the region information of the target object within the group based on the group information and intra-group constraints includes: When the constraint relationship within the group is the size consistency constraint, the original size of each target object within the group is determined based on the region information of each target object within the group; Based on the original size, determine the scaling factor for each target object within the group; The region information of each target object within the group is updated based on the scaling factor.

3. The method as described in claim 2, characterized in that, The original size includes the original vertical size and the original horizontal size, and the scaling factor includes the horizontal scaling factor and the vertical scaling factor; Determining the scaling factor for each target object within the group based on the original size includes: Based on the original horizontal dimensions, determine the target horizontal dimension of each target object within the group; based on the original vertical dimensions, determine the target vertical dimension of each target object within the group. The horizontal scaling factor is obtained by dividing the target horizontal dimension of each target object in the group by the original horizontal dimension; and the vertical scaling factor is obtained by dividing the target vertical dimension of each target object in the group by the original vertical dimension.

4. The method as described in claim 1, characterized in that, The step of adjusting the region information of the target object within the group based on the group information and intra-group constraints includes: When the intra-group constraint relationship is the position consistency constraint, the original reference point position of each target object in the group is determined based on the region information of each target object in the group; Based on the original reference point position of each target object in the group, determine the target reference point position of each target object in the group; Based on the original reference point position and the target reference point position of each target object in the group, the regional information of each target object in the group is updated.

5. The method as described in claim 4, characterized in that, Determining the target reference point position of each target object in the group based on the original reference point position of each target object in the group includes: A linear regression is performed on the original reference point position of the target object within the group to obtain the linear regression equation; Based on the linear regression equation, the target reference point position of each target object within the group is determined.

6. The method according to any one of claims 1-5, characterized in that, The step of grouping the at least two target objects into at least one group based on the region information and the category information includes: Based on the region information and the category information, at least two target objects in the first image are clustered to obtain at least one group; Determine the within-class variance of each group in the at least one group; When the intra-class variance of the at least one group is less than a preset threshold, the target objects contained in each group are used as the group information of the at least one group.

7. The method according to any one of claims 1-6, characterized in that, The step of identifying the target object within the group based on the adjusted region information includes: Based on the category information and the adjusted region information, the target object within the group is identified.

8. An image processing apparatus, characterized in that, The image processing apparatus includes a processor and a memory, the memory being used to store a computer program, and the processor being used to invoke the computer program stored in the memory, so that the electronic device implements the method according to any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when run on a computer, causes the computer to perform the method as described in any one of claims 1-7.

Images