Multi-camera-based image recognition method, device, equipment and storage medium

Abstract

The application discloses a kind of based on multi-camera image recognition method, device, equipment and storage medium, the method is extracted target feature from image feature by based on the position of detection frame in sub-sample, obtain target feature group, with each detection frame as input, target feature group is deep learning, obtain the object recognition network of cross camera, obtain the scene image to be identified photographed by multiple cameras, and sequentially input to object recognition network for object recognition, obtain the object recognition result in each scene image, each object recognition result is compared with each other, determine the same object in each scene image based on the result of comparison.The same object in the image of cross camera is identified based on the object recognition network, solve the problem that the identification process of the same object is complex in the multi-camera image perception technology, leading to the problem of low efficiency of knowledge identification.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and in particular to an image recognition method, apparatus, device, and storage medium based on multiple cameras. Background Technology

[0002] With the development of artificial intelligence technology, especially in the research of autonomous driving, vehicles can automatically identify objects by installing hardware such as LiDAR and image acquisition on them, and then using image analysis software programs, thus achieving driverless driving.

[0003] Autonomous vehicles need to achieve 360-degree, blind-spot-free perception of surrounding obstacles. This is mainly achieved by capturing images from cameras in different directions and then merging these images to achieve 360-degree perception. However, current solutions for identifying the same object in images captured by multiple cameras primarily rely on recognizing the object's coordinates or feature similarity across different images. Since the coordinates of the same object can differ across different cameras, recognizing the same object in multiple images requires coordinate transformation or adjustment, increasing the difficulty of image recognition algorithms and making them overly complex. Therefore, multi-camera collaboration is needed to better address this cross-camera object recognition problem. Thus, there is an urgent need for a solution that can quickly identify the same object across multiple cameras. Summary of the Invention

[0004] The main objective of this invention is to solve the problem of low perception and recognition efficiency caused by the complex recognition process for the same object in multi-camera image perception technology.

[0005] The first aspect of this invention provides an image recognition method based on multiple cameras, the image recognition method comprising:

[0006] Acquire training samples, wherein the training samples include at least two sub-samples, and each sub-sample includes at least two driving scene images captured by different cameras;

[0007] Extract object features from each driving scene image in each of the sub-samples to obtain multi-scale image features;

[0008] Based on the position of the detection box in the sub-sample, target features are extracted from the image features to obtain a target feature group;

[0009] Using each detection box as input, deep learning is performed on the corresponding target feature groups to obtain a cross-camera object recognition network;

[0010] Multiple scene images captured by cameras are acquired and sequentially input into the object recognition network for object recognition, thereby obtaining the object recognition results in each scene image;

[0011] The object recognition results are compared with each other, and the same objects in each scene image are determined based on the comparison results.

[0012] Optionally, in a first implementation of the first aspect of the present invention, the step of extracting object features from each driving scene image in each of the sub-samples to obtain multi-scale image features includes:

[0013] A batch of subsamples is drawn from the training samples, wherein the batch of subsamples includes at least N subsamples, where N is a natural number greater than 1;

[0014] Using convolutional neural networks and feature pyramid networks, features are extracted from the driving scene images in each of the subsamples, and multi-scale image features are generated based on the extracted features.

[0015] Optionally, in a first implementation of the second aspect of the present invention, the step of extracting target features from the image features based on the position of the detection box in the sub-sample to obtain a target feature group includes:

[0016] Extract the detection bounding boxes of each camera in the batch of sub-samples, and determine the position of the detection bounding box in the corresponding image;

[0017] Based on the position of each detection box, the corresponding target features are extracted from the image features, and the target features belonging to the same subsample are grouped into the same category to obtain the corresponding target feature group.

[0018] Optionally, in a third implementation of the first aspect of the present invention, the step of using each detection box as input to perform deep learning on the corresponding target feature group to obtain a cross-camera object recognition network includes:

[0019] All detection boxes at the same location are obtained and used as query commands in sequence to query the target feature group corresponding to the location, wherein the target feature group is a multi-dimensional tensor;

[0020] The multi-dimensional tensor is input into a multi-layer network corresponding to the self-attention mechanism. The multi-layer network learns and identifies the information and the relationships between the information in each dimension of the tensor. The learned content in each dimension is used to adjust the correlation between the layers of the multi-layer network to obtain a cross-camera object recognition network.

[0021] Optionally, in a fourth implementation of the first aspect of the present invention, after performing deep learning on the corresponding target feature groups using each detection box as input to obtain a cross-camera object recognition network, the method further includes:

[0022] Select at least one sub-sample from the training samples to construct the validation samples;

[0023] Using the object recognition network, each image in the verification sample is identified, and a verification tensor is output.

[0024] The verification tensor is matched with the ground truth value corresponding to the verification sample, and a loss function is constructed based on the matching result;

[0025] The object recognition network is optimized based on the loss function.

[0026] Optionally, in a fifth implementation of the first aspect of the present invention, the step of acquiring scene images to be identified captured by multiple cameras and sequentially inputting them into the object recognition network for object recognition to obtain object recognition results in each scene image includes:

[0027] Collect scene images to be identified from multiple cameras;

[0028] Multiple scene images are sequentially input into the object recognition network, which extracts features based on the detection box positions corresponding to the scene images to obtain the target image features in each scene image;

[0029] The target image features extracted from each scene image are spliced ​​together to obtain the object recognition result.

[0030] Optionally, in a sixth implementation of the first aspect of the present invention, the step of concatenating and combining the target image features extracted from each scene image to obtain the object recognition result includes:

[0031] The target image features extracted from each scene image are converted into multiple parameter representations;

[0032] By combining the parameter representations corresponding to the scene images, we can obtain the parameter representations of the objects.

[0033] Convolution operations are performed based on the object's parameter representation to obtain the object's multidimensional tensor.

[0034] A second aspect of the present invention provides an image recognition device based on multiple cameras, the image recognition device based on multiple cameras comprising:

[0035] An acquisition module is used to acquire training samples, wherein the training samples include at least two sub-samples, and each sub-sample includes at least two driving scene images captured by different cameras;

[0036] An extraction module is used to extract object features from each driving scene image in each of the sub-samples to obtain multi-scale image features; and to extract target features from the image features based on the position of the detection box in the sub-sample to obtain a target feature group.

[0037] The training module is used to perform deep learning on the corresponding target feature groups with each detection box as input, so as to obtain a cross-camera object recognition network;

[0038] The recognition module is used to acquire scene images to be recognized captured by multiple cameras, and input them sequentially into the object recognition network for object recognition to obtain object recognition results in each scene image;

[0039] The comparison module is used to compare the object recognition results of each object and determine the same objects in each scene image based on the comparison results.

[0040] Optionally, in a first implementation of the second aspect of the present invention, the acquisition module includes:

[0041] A sampling unit is used to extract a batch of subsamples from the training samples, wherein the batch of subsamples includes at least N subsamples, where N is a natural number greater than 1;

[0042] The generation unit is used to extract features from the driving scene image in each of the subsamples using a convolutional neural network and a feature pyramid network, and to generate multi-scale image features based on the extracted features.

[0043] Optionally, in a second implementation of the second aspect of the present invention, the extraction module includes:

[0044] The location extraction unit is used to extract the detection boxes of each camera in the batch of sub-samples and determine the position of the detection boxes in the corresponding images;

[0045] The extraction unit is used to extract the corresponding target features from the image features based on the position of each detection box, and to group the target features belonging to the same subsample into the same category to obtain the corresponding target feature group.

[0046] Optionally, in a third implementation of the second aspect of the present invention, the identification module includes:

[0047] The query unit is used to obtain all detection boxes at the same location and sequentially use them as query commands to query the target feature group corresponding to the location, wherein the target feature group is a multi-dimensional tensor;

[0048] The recognition unit is used to input the multi-dimensional tensor into a multi-layer network corresponding to the self-attention mechanism. The multi-layer network learns and recognizes the information and the relationships between the information in each dimension of the tensor, and adjusts the correlation between the layers of the multi-layer network based on the learned content in each dimension, thereby obtaining a cross-camera object recognition network.

[0049] Optionally, in a fourth implementation of the second aspect of the present invention, the multi-camera-based image recognition device further includes an optimization module, which is specifically used for:

[0050] Select at least one sub-sample from the training samples to construct the validation samples;

[0051] Using the object recognition network, each image in the verification sample is identified, and a verification tensor is output.

[0052] The verification tensor is matched with the ground truth value corresponding to the verification sample, and a loss function is constructed based on the matching result;

[0053] The object recognition network is optimized based on the loss function.

[0054] Optionally, in a fifth implementation of the second aspect of the present invention, the identification unit is specifically used for:

[0055] Collect scene images to be identified from multiple cameras;

[0056] Multiple scene images are sequentially input into the object recognition network, which extracts features based on the detection box positions corresponding to the scene images to obtain the target image features in each scene image;

[0057] The target image features extracted from each scene image are spliced ​​together to obtain the object recognition result.

[0058] Optionally, in a sixth implementation of the second aspect of the present invention, the comparison module includes:

[0059] The transformation unit is used to convert the target image features extracted from each scene image into multiple parameter representations;

[0060] The combination unit is used to combine the parameter representations corresponding to the scene images to obtain the parameter representations of the object;

[0061] The computation unit is used to perform convolution operations based on the object's parameter representation to obtain the object's multidimensional tensor.

[0062] A third aspect of the present invention provides a computer device, comprising: a memory and at least one processor, wherein the memory stores instructions, and the memory and the at least one processor are interconnected via a circuit; the at least one processor invokes the instructions in the memory to cause the computer device to perform the steps of the above-described multi-camera-based image recognition method.

[0063] A fourth aspect of the present invention provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the steps of the above-described multi-camera-based image recognition method.

[0064] The beneficial effects achieved by this application are:

[0065] This application extracts object features from driving scene images in each subsample of the training samples to obtain multi-scale image features. Based on the position of the detection boxes in the subsamples, target features are extracted from the image features to obtain target feature groups. Using each detection box as input, a self-attention mechanism is used to perform deep learning on the corresponding target feature groups to obtain a cross-camera object recognition network. Multiple scene images to be recognized are acquired from multiple cameras and sequentially input into the object recognition network for object recognition. The object recognition results in each scene image are obtained, and the recognition results are compared to determine the same objects in each scene image. Based on this object recognition network, the same objects in images from different cameras can be recognized without individually recognizing each camera. This improves the ability, accuracy, and efficiency of autonomous vehicles to recognize objects or obstacles, while also enhancing the user experience of autonomous driving. Attached Figure Description

[0066] Figure 1 This is a schematic diagram of the first embodiment of the image recognition method based on multiple cameras in this invention.

[0067] Figure 2 This is a schematic diagram of a second embodiment of the image recognition method based on multiple cameras in this invention.

[0068] Figure 3 This is a schematic diagram of a third embodiment of the image recognition method based on multiple cameras in this invention.

[0069] Figure 4 This is a schematic diagram of one embodiment of the image recognition device based on multiple cameras according to the present invention;

[0070] Figure 5 This is a schematic diagram of another embodiment of the image recognition device based on multiple cameras in this invention;

[0071] Figure 6 This is a schematic diagram of one embodiment of the computer device in this invention. Detailed Implementation

[0072] This invention provides an image recognition method, apparatus, device, and storage medium based on multiple cameras. By proposing a novel deep neural network design, it can obtain consistent obstacle boxes and instance segmentation results across different cameras. The same object has the same ID in different cameras. The network directly outputs the detection results of each object in all cameras.

[0073] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” or “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0074] For ease of understanding, the specific process of the embodiments of the present invention is described below. Please refer to [link / reference]. Figure 1 The first embodiment of the multi-camera image recognition method in this invention is applied to an autonomous driving system. The autonomous driving system includes at least an in-vehicle system and a camera group. Each camera in the camera group collects environmental information of the surrounding images of each location the vehicle passes through, and then inputs them into the in-vehicle system for image recognition analysis. Specifically, the image recognition method includes the following steps:

[0075] 101, Obtain training samples;

[0076] In this embodiment, the training samples include at least two sub-samples, and each sub-sample includes at least two driving scene images captured by different cameras;

[0077] Specifically, the training samples can be extracted from vehicle road test data or directly from vehicle driving records. After extracting the video or images of the driving scenario, these data are used to create samples. In practical applications, the videos or images are labeled using a manually defined method. Specifically, this labeling involves marking regions in the video or image that belong to the same object, thereby obtaining the training samples.

[0078] In this embodiment, defining training samples specifically includes defining model training samples and validation samples. In practical applications, training samples can also be acquired using multiple cameras on the vehicle. These cameras can be controlled to capture images of the same object from their respective angles, and the images captured by each camera are then correlated, along with the detection boxes, to obtain the training samples.

[0079] 102. Extract object features from each driving scene image in each subsample to obtain multi-scale image features;

[0080] In this step, a training sample includes at least two sub-samples, and each sub-sample includes at least two images, which are images of the same object from different angles. In practical applications, if there are many training samples, different numbers of sub-samples are extracted from the training samples using an extraction algorithm as model training samples and validation samples. Then, different driving scene images are extracted according to the detection boxes labeled as the same object. Finally, the features extracted from different images are combined to construct multi-scale image features.

[0081] In practical applications, convolutional neural networks are used to extract features from images. The extraction process mainly involves converting the entire driving scene image into a feature map. Specifically, convolutional neural networks are used to extract the pixels belonging to the road in the driving scene image to obtain the road image. Then, different objects in the road image are depicted by contour drawing and binarized to obtain image features. Finally, the image features from each angle are combined and stitched together to obtain multi-scale image features.

[0082] 103. Based on the position of the detection box in the subsample, target features are extracted from the image features to obtain the target feature group;

[0083] In this embodiment, after converting the images in the subsamples into image features, the detection boxes of each image are obtained, and the detection boxes belonging to the same object in each image are filtered out. The target features corresponding to the detection boxes are extracted from the corresponding image features in turn. Then, the target feature groups extracted from different image features are combined according to the camera's order to obtain the target feature combination, which is the complete image feature corresponding to the object.

[0084] 104. Using each detection box as input, deep learning is performed on the corresponding target feature groups to obtain a cross-camera object recognition network;

[0085] In this embodiment, after constructing the overall image features of the object based on the above method, the recognition process of the cross-camera object recognition network is executed. That is, after extracting the image features in the training samples, the detection box in the training samples is randomly selected, and then the corresponding target feature group is queried based on the detection box. The target feature group is learned by the neural network to obtain the feature expression of the object from different angles, thereby obtaining the object recognition network.

[0086] In practical applications, a self-attention mechanism is used to determine the corresponding target feature group with the detection box as input. Then, learning is performed based on the target feature group to obtain the corresponding target feature group expression rules. Based on these target rules, a corresponding object recognition network is constructed to achieve cross-camera image recognition.

[0087] 105. Acquire scene images to be identified captured by multiple cameras, and input them sequentially into the object recognition network for object recognition to obtain object recognition results in each scene image;

[0088] In this step, multiple cameras are installed on the vehicle to acquire scene images of the current scene from different angles. The acquired scene images are then input into an object recognition network, which identifies the objects in the scene images and outputs the recognition results. The recognition results are feature groups of the objects, which include image features and the relationships between image features.

[0089] 106. Compare the object recognition results of each object with each other, and determine the same objects in each scene image based on the comparison results.

[0090] In this step, the output results in each scene image are compared. If the comparison results are consistent, it means that the objects identified in each scene image belong to the same object.

[0091] Furthermore, after identifying objects belonging to the same category, these objects are labeled, or images from multiple cameras are fused based on the comparison results to obtain the final image.

[0092] In summary, by extracting object features from each driving scene image in each subsample of the training samples, multi-scale image features are obtained. Based on the position of the detection boxes in the subsamples, target features are extracted from the image features to obtain target feature groups. Using each detection box as input, a self-attention mechanism is used to perform deep learning on the corresponding target feature groups to obtain a cross-camera object recognition network. Multiple scene images to be recognized are acquired from multiple cameras and sequentially input into the object recognition network for object recognition, obtaining object recognition results in each scene image. The object recognition results are compared, and the same objects in each scene image are identified based on the comparison results. This provides a cross-camera object recognition network that can recognize objects from multiple cameras. This network comprehensively recognizes image information from each camera, avoiding inconsistencies in recognition information and solving the problem of complex recognition processes and low recognition efficiency for the same object in multi-camera image perception technology.

[0093] Please see Figure 2 and 3 The second embodiment of the image recognition method based on multiple cameras provided in this invention includes:

[0094] 201, Obtain training samples;

[0095] 202, Draw a batch of subsamples from the training samples;

[0096] In this step, a batch of subsamples includes at least N subsamples, where N is a natural number greater than 1. Specifically, a certain number of subsamples are extracted from the training samples using a random sampling algorithm to construct the network training samples for the object recognition network. The number of subsamples extracted can be set as a percentage or as a fixed value.

[0097] In practical applications, random sampling algorithms are used to extract sub-samples from the training samples, corresponding to specific scenarios such as obstacle avoidance and intersections. These sub-samples are then used to train the network. Furthermore, different proportions of scenarios can be used to extract sub-samples, creating a mixed network training sample.

[0098] 203. Using convolutional neural networks and feature pyramid networks, features are extracted from driving scene images in each subsample, and multi-scale image features are generated based on the extracted features.

[0099] In this embodiment, a combination of residual neural network and feature pyramid network in convolutional neural network is used to extract features from each driving scene image in the subsample. Each subsample contains images of the same object from different angles.

[0100] This involves using convolutional neural networks (such as residual neural networks) and feature pyramid networks to extract multi-scale image features F. For each training sample, there are M input images, and the feature size is N*M*H*W*C, where N is the batch size, M is the number of images in each sample, H is the height, W is the width, and C is the number of channels.

[0101] 204. Extract the detection bounding boxes of each camera in the batch sub-samples and determine the position of the detection bounding box in the corresponding image;

[0102] In this step, detection boxes are determined for each image in the extracted sub-samples of the corresponding batch. Further, if multiple detection boxes exist in each image, the detection boxes representing the same object in the sub-sample are selected based on their annotation information. Preferably, this is determined based on coordinate information; that is, the position of the object in each image is determined by its three-dimensional coordinates, thereby determining the position of the detection box. The position of the detection box can be directly determined based on the correspondence between the detection box, the object, and its three-dimensional coordinates. The position of the detection box is then used to determine whether it corresponds to the same object.

[0103] 205. Based on the position of each detection box, extract the corresponding target features from the image features, and group the target features belonging to the same subsample into the same category to obtain the corresponding target feature group;

[0104] In this embodiment, the image features here include the features expressed by an object at different angles (i.e., different cameras). The location for extracting features from the image features is determined by the detection box corresponding to the object in each camera, and the target features are extracted based on the location to generate a target feature group.

[0105] Specifically, using the results of the individual detection boxes from the input multi-camera system, the corresponding image features are extracted from the image features F output in the above steps according to the position of the detection boxes. Consider a total of L input detection boxes, and the final tensor dimension is N*L*C.

[0106] 206. Using each detection box as input, a self-attention mechanism is used to perform deep learning on the corresponding target feature groups to obtain a cross-camera object recognition network.

[0107] In this embodiment, all detection boxes at the same location are acquired and sequentially used as query commands to query the target feature group corresponding to the location. The target feature group is a multi-dimensional tensor. The multi-dimensional tensor is input into a multi-layer network corresponding to the self-attention mechanism. The multi-layer network learns and recognizes the information and the relationship between the information in each dimension of the tensor. The learned content in each dimension is used to adjust the correlation of each layer of the multi-layer network to obtain a cross-camera object recognition network.

[0108] Specifically, after extracting the target feature group, the process also includes establishing a corresponding relationship between the target feature group and the detection boxes of each camera. During training, the detection boxes are used as input parameters to select the corresponding target feature group from the target feature group and input it into a preset network, such as a self-attention mechanism. The network learns the target features in the target feature group and the relationship between the target features to construct an object recognition network.

[0109] In this embodiment, to improve the accuracy of the object recognition network, after training, it further includes:

[0110] Select at least one sub-sample from the training samples to construct the validation samples;

[0111] Using the object recognition network, each image in the verification sample is identified, and a verification tensor is output.

[0112] The verification tensor is matched with the ground truth value corresponding to the verification sample, and a loss function is constructed based on the matching result;

[0113] The object recognition network is optimized based on the loss function.

[0114] Since each input detection box corresponds to a set of full-camera output detection results, all full-camera output detection results are considered as a set. The full-camera ground truth boxes are also a set. Therefore, the distance between the sets of full-camera detection boxes is used as the loss function. Specifically, for any pair of full-camera predicted boxes and full-camera ground truth boxes, the loss function includes the classification loss l. cls , center position regression loss l center Size regression loss l siXe Next, the Hungarian algorithm is used to calculate the minimum loss among all possible pairings, which is then used as the final loss function to supervise the training of the neural network.

[0115] 207. Acquire scene images to be identified captured by multiple cameras and input them sequentially into the object recognition network for object recognition to obtain object recognition results in each scene image;

[0116] In this embodiment, multiple cameras capture images of the scene to be identified.

[0117] Multiple scene images are sequentially input into the object recognition network, which extracts features based on the detection box positions corresponding to the scene images to obtain the target image features in each scene image;

[0118] The target image features extracted from each scene image are spliced ​​together to obtain the object recognition result.

[0119] Specifically, an N*L*C dimensional tensor is fed into a multi-layer self-attention network to complete information exchange. The output of the multi-layer self-attention network is also an N*L*C dimensional tensor. Next, consider that for any object, it needs to be represented by 2+2+H+1=H+5 parameters on each camera. Among them, the center point position has 2 parameters, the size has 2 parameters, the categories have H, and whether it is an object has 1 parameter. Therefore, to represent the position of an object on M cameras, M*(H+5) parameters are needed. Therefore, a 1*1 convolution is used to transform the N*L*C dimensional tensor into an N*L*(M*(H+5)) dimensional tensor, and the N*L*(M*(H+5)) dimensional tensor is used as the final network prediction output.

[0120] Furthermore, the step of concatenating and combining the target image features extracted from each scene image to obtain the object recognition result includes:

[0121] The target image features extracted from each scene image are converted into multiple parameter representations;

[0122] By combining the parameter representations corresponding to the scene images, we can obtain the parameter representations of the objects.

[0123] Convolution operations are performed based on the object's parameter representation to obtain the object's multidimensional tensor.

[0124] 208. Compare the object recognition results with each other, and determine the same objects in each scene image based on the comparison results.

[0125] In this embodiment, the input to the neural network is information from all cameras and the individual obstacle detection results for each image. The obstacle detection bounding boxes from each camera are used as queries and correlated within the transformer. Finally, a consistent obstacle output across all cameras is decoded. Finally, Hungarian matching is performed between these obstacle predictions and the ground truth values, and the loss function under the optimal matching is calculated. This loss function is used as the final loss function of the network to supervise network training.

[0126] In summary, by designing a neural network architecture, it is possible to directly predict the consistent detection results of each obstacle across all cameras, and at the same time, to obtain complete obstacle information across cameras, which greatly improves recognition efficiency and accuracy.

[0127] The above describes the image recognition method based on multiple cameras provided in the embodiments of the present invention. The following describes the image recognition device based on multiple cameras in the embodiments of the present invention. Please refer to [link to relevant documentation]. Figure 4 The image recognition device based on multiple cameras according to embodiments of the present invention includes:

[0128] The acquisition module 401 is used to acquire training samples, wherein the training samples include at least two sub-samples, and each sub-sample includes at least two driving scene images captured by different cameras;

[0129] Extraction module 402 is used to extract object features from each driving scene image in each of the sub-samples to obtain multi-scale image features; and to extract target features from the image features based on the position of the detection box in the sub-sample to obtain a target feature group;

[0130] Training module 403 is used to perform deep learning on the corresponding target feature groups using a self-attention mechanism with each detection box as input, so as to obtain a cross-camera object recognition network.

[0131] The recognition module 404 is used to acquire scene images to be recognized captured by multiple cameras, and input them sequentially into the object recognition network for object recognition to obtain object recognition results in each scene image;

[0132] The comparison module 405 is used to compare the object recognition results with each other and determine the same objects in each scene image based on the comparison results.

[0133] In this embodiment of the invention, target features are extracted from image features based on the position of detection boxes in sub-samples to obtain a target feature group. Using each detection box as input, deep learning is performed on the target feature group to obtain a cross-camera object recognition network. Multiple scene images to be recognized are acquired from various cameras and sequentially input into the object recognition network for object recognition. Object recognition results are obtained for each scene image. These results are compared, and the same objects in each scene image are identified based on the comparison results. This object recognition network enables the recognition of the same object in images from multiple cameras, solving the problem of complex recognition processes and low recognition efficiency in multi-camera image perception technology.

[0134] Please see Figure 5 The second embodiment of the multi-camera-based image recognition device provided in this invention specifically includes:

[0135] The acquisition module 401 is used to acquire training samples, wherein the training samples include at least two sub-samples, and each sub-sample includes at least two driving scene images captured by different cameras;

[0136] Extraction module 402 is used to extract object features from each driving scene image in each of the sub-samples to obtain multi-scale image features; and to extract target features from the image features based on the position of the detection box in the sub-sample to obtain a target feature group;

[0137] Training module 403 is used to perform deep learning on the corresponding target feature groups using a self-attention mechanism with each detection box as input, so as to obtain a cross-camera object recognition network.

[0138] The recognition module 404 is used to acquire scene images to be recognized captured by multiple cameras, and input them sequentially into the object recognition network for object recognition to obtain object recognition results in each scene image;

[0139] The comparison module 405 is used to compare the object recognition results with each other and determine the same objects in each scene image based on the comparison results.

[0140] In this embodiment, the acquisition module 401 includes:

[0141] Sampling unit 4011 is used to extract a batch of subsamples from the training samples, wherein the batch of subsamples includes at least N subsamples, where N is a natural number greater than 1;

[0142] The generation unit 4012 is used to extract features from the driving scene image in each of the subsamples using a convolutional neural network and a feature pyramid network, and to generate multi-scale image features based on the extracted features.

[0143] In this embodiment, the extraction module 402 includes:

[0144] The location extraction unit 4021 is used to extract the detection boxes of each camera in the batch of sub-samples and determine the position of the detection boxes in the corresponding images;

[0145] The extraction unit 4022 is used to extract the corresponding target features from the image features based on the position of each detection box, and to group the target features belonging to the same subsample into the same category to obtain the corresponding target feature group.

[0146] In this embodiment, the identification module 404 includes:

[0147] The query unit 4041 is used to obtain all detection boxes at the same location and sequentially use them as query commands to query the target feature group corresponding to the location, wherein the target feature group is a multi-dimensional tensor;

[0148] The recognition unit 4042 is used to input the multi-dimensional tensor into the multi-layer network corresponding to the self-attention mechanism. The multi-layer network learns and recognizes the information and the relationship between the information in each dimension of the tensor, and adjusts the correlation of each layer of the multi-layer network based on the learned content in each dimension, thereby obtaining a cross-camera object recognition network.

[0149] In this embodiment, the multi-camera-based image recognition device further includes an optimization module 406, which is specifically used for:

[0150] Select at least one sub-sample from the training samples to construct the validation samples;

[0151] Using the object recognition network, each image in the verification sample is identified, and a verification tensor is output.

[0152] The verification tensor is matched with the ground truth value corresponding to the verification sample, and a loss function is constructed based on the matching result;

[0153] The object recognition network is optimized based on the loss function.

[0154] In this embodiment, the identification unit 4042 is specifically used for:

[0155] Collect scene images to be identified from multiple cameras;

[0156] Multiple scene images are sequentially input into the object recognition network, which extracts features based on the detection box positions corresponding to the scene images to obtain the target image features in each scene image;

[0157] The target image features extracted from each scene image are spliced ​​together to obtain the object recognition result.

[0158] In this embodiment, the comparison module 405 includes:

[0159] The conversion unit 4051 is used to convert the target image features extracted from each scene image into multiple parameter expressions;

[0160] The combination unit 4052 is used to combine the parameter expressions corresponding to the scene images to obtain the parameter expressions of the object;

[0161] The computation unit 4053 is used to perform convolution operations based on the object's parameter representation to obtain the object's multidimensional tensor.

[0162] This embodiment, based on the previous embodiment, adds other functional modules. Through these modules, the device extracts object features from each driving scene image in each sub-sample of the training samples to obtain multi-scale image features. Based on the position of the detection box in the sub-sample, target features are extracted from the image features to obtain target feature groups. Using each detection box as input, a self-attention mechanism is used to perform deep learning on the corresponding target feature groups to obtain a cross-camera object recognition network. Multiple scene images to be recognized are acquired from multiple cameras and sequentially input into the object recognition network for object recognition, obtaining object recognition results in each scene image. The object recognition results are compared, and the same objects in each scene image are determined based on the comparison results. Based on this object recognition network, the same objects in images from multiple cameras can be recognized without individually recognizing each camera, thus improving the autonomous vehicle's ability, accuracy, and efficiency in recognizing objects or obstacles, and also enhancing the user experience of autonomous driving.

[0163] above Figure 4 and Figure 5 The image recognition device based on multiple cameras in this embodiment of the invention will be described in detail from the perspective of modular functional entities. The computer device in this embodiment of the invention will be described in detail from the perspective of hardware processing.

[0164] Figure 6 This is a schematic diagram of a computer device 600 provided in an embodiment of the present invention. The computer device 600 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 610 (e.g., one or more processors) and a memory 620, and one or more storage media 630 (e.g., one or more mass storage devices) for storing application programs 633 or data 632. The memory 620 and storage media 630 can be temporary or persistent storage. The program stored in the storage media 630 may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the computer device 600. Furthermore, the processor 610 may be configured to communicate with the storage media 630 and execute the series of instruction operations in the storage media 630 on the computer device 600 to implement the steps of the multi-camera-based image recognition method described above.

[0165] Computer device 600 may also include one or more power supplies 640, one or more wired or wireless network interfaces 650, one or more input / output interfaces 660, and / or one or more operating systems 631, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will understand that... Figure 6 The computer device structure shown does not constitute a limitation on the computer device provided in this application. It may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0166] A third aspect of the present invention provides a computer device, comprising: a memory and at least one processor, wherein the memory stores instructions, and the memory and the at least one processor are interconnected via a circuit; the at least one processor invokes the instructions in the memory to cause the computer device to perform the steps of the multi-camera-based image recognition method described above.

[0167] The present invention also provides a computer-readable storage medium, which can be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium, wherein the computer-readable storage medium stores instructions that, when the instructions are executed on a computer, cause the computer to perform the steps of the multi-camera-based image recognition method.

[0168] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system, device, or unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0169] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0170] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-camera based image recognition method, characterized in that, The image recognition method includes: Acquire training samples, wherein the training samples include at least two sub-samples, and each sub-sample includes at least two driving scene images captured by different cameras; Extract object features from each driving scene image in each of the sub-samples to obtain multi-scale image features; Based on the position of the detection box in the sub-sample, target features are extracted from the image features to obtain a target feature group; Using each detection box as input, deep learning is performed on the corresponding target feature groups to obtain a cross-camera object recognition network; Based on the training samples, validation samples are constructed. Based on the validation samples and the detection results output by the object recognition network, sets are constructed respectively. Based on the constructed sets, the Hungarian algorithm is used to calculate the minimum loss among all possible pairing forms as the final loss function used to supervise the training of the object recognition network. Multiple scene images captured by cameras are acquired and sequentially input into the object recognition network for object recognition, thereby obtaining the object recognition results in each scene image; The object recognition results are compared with each other, and the same objects in each scene image are determined based on the comparison results.

2. The multi-camera based image recognition method of claim 1, wherein, The step of extracting object features from each driving scene image in each of the sub-samples to obtain multi-scale image features includes: A batch of subsamples is drawn from the training samples, wherein the batch of subsamples includes at least N subsamples, where N is a natural number greater than 1; Feature extraction is performed on the driving scene images in each of the subsamples, and multi-scale image features are generated based on the extracted features.

3. The multi-camera based image recognition method of claim 2, wherein, The step of extracting target features from the image features based on the position of the detection box in the sub-sample to obtain a target feature group includes: Extract the detection bounding boxes of each camera in the batch of sub-samples, and determine the position of the detection bounding box in the corresponding image; Based on the position of each detection box, the corresponding target features are extracted from the image features, and the target features belonging to the same subsample are grouped into the same category to obtain the corresponding target feature group.

4. The multi-camera based image recognition method according to any one of claims 1-3, wherein, The process of using each detection box as input to perform deep learning on the corresponding target feature groups to obtain a cross-camera object recognition network includes: All detection boxes at the same location are obtained and used as query commands in sequence to query the target feature group corresponding to the location, wherein the target feature group is a multi-dimensional tensor; By learning and recognizing information and relationships between information in each dimension of the tensor through a multi-layer network, and adjusting the correlation between the layers of the multi-layer network based on the learned content in each dimension, a cross-camera object recognition network is obtained.

5. The multi-camera based image recognition method of claim 4, wherein, The process of constructing validation samples based on the training samples, constructing sets based on the validation samples and the detection results output by the object recognition network from the validation samples, and using the Hungarian algorithm to calculate the minimum loss among all possible pairings based on the constructed sets as the final loss function used to supervise the training of the object recognition network includes: Select at least one sub-sample from the training samples to construct the validation samples; Using the object recognition network, each image in the verification sample is identified, and a verification tensor is output. The verification tensor is matched with the ground truth value corresponding to the verification sample, and a loss function is constructed based on the matching result; The object recognition network is optimized based on the loss function.

6. The multi-camera based image recognition method of claim 5, wherein, The process involves acquiring scene images captured by multiple cameras and sequentially inputting them into the object recognition network for object recognition, thereby obtaining object recognition results in each scene image, including: Collect scene images to be identified from multiple cameras; Multiple scene images are sequentially input into the object recognition network, which extracts features based on the detection box positions corresponding to the scene images to obtain the target image features in each scene image; The target image features extracted from each scene image are spliced ​​together to obtain the object recognition result.

7. The multi-camera based image recognition method of claim 6, wherein, The step of concatenating and combining the target image features extracted from each scene image to obtain the object recognition result includes: The target image features extracted from each scene image are converted into multiple parameter representations; By combining the parameter representations corresponding to the scene images, we can obtain the parameter representations of the objects. Convolution operations are performed based on the object's parameter representation to obtain the object's multidimensional tensor.

8. A multi-camera based image recognition apparatus, characterized by, The multi-camera-based image recognition device includes: An acquisition module is used to acquire training samples, wherein the training samples include at least two sub-samples, and each sub-sample includes at least two driving scene images captured by different cameras; An extraction module is used to extract object features from each driving scene image in each of the sub-samples to obtain multi-scale image features; and to extract target features from the image features based on the position of the detection box in the sub-sample to obtain a target feature group. The training module is used to perform deep learning on the corresponding target feature groups with each detection box as input, so as to obtain a cross-camera object recognition network; An optimization module is used to construct validation samples based on the training samples, construct sets based on the validation samples and the detection results output by the object recognition network, and use the Hungarian algorithm to calculate the minimum loss among all possible pairing forms based on the constructed sets as the final loss function used to supervise the training of the object recognition network. The recognition module is used to acquire scene images to be recognized captured by multiple cameras, and input them sequentially into the object recognition network for object recognition to obtain object recognition results in each scene image; The comparison module is used to compare the object recognition results of each object and determine the same objects in each scene image based on the comparison results.

9. A computer device, characterized in that, The computer device includes: a memory and at least one processor, wherein the memory stores instructions, and the memory and the at least one processor are interconnected via a line; The at least one processor invokes the instructions in the memory to cause the computer device to perform the steps of the multi-camera-based image recognition method as described in any one of claims 1-7.

10. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by the processor, it implements the various steps of the multi-camera-based image recognition method as described in any one of claims 1-7.

Images