Object recognition method, device, electronic device and storage medium
By splitting the neural network model into two parts: feature extraction and object recognition, using the first neural network to extract features and determine the second neural network for recognition based on the recognition task and size, the large number of models and long training time in the prior art are solved. The problem of this paper improves the utilization rate of computing resources and the scope of application of the model.
Patent Information
- Application Number
- CN202210072098.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-01-21
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2042-01-21
AI Technical Summary
In the prior art, neural network models for object recognition need to be independently constructed and trained for each recognition task and each input size, resulting in a large number of models, a long training time and a low computing resource utilization rate.
Split the neural network model into a first neural network for feature extraction and a second neural network for object recognition. The first neural network performs feature extraction on the input image, and according to the size of the recognition task and feature map Determine the corresponding second neural network for object recognition.
The number of neural network models is reduced, training time is reduced, the utilization rate of computing resources is improved, and the scope of application of neural network models is expanded.
Smart Images

Figure CN114495039B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence, and more specifically, to an object recognition method, device, electronic device and storage medium. Background Art
[0002] With the development of computer vision technology and autonomous driving technology, multiple cameras are usually used to identify the environment and objects around the vehicle. In related technologies, images are usually input into a neural network model, and the neural network model outputs the recognition results. In related technologies, the neural network model used for object recognition is not only related to the recognition task, but also has requirements on the input size. Therefore, it is necessary to pre-build a corresponding neural network model for each recognition task and each input size. Therefore, in scenarios where there are multiple recognition tasks with different sizes, there are more neural network models that need to be built and trained, and the time for training the neural network model is also longer. Summary of the invention
[0003] In view of this, the embodiments of the present application propose an object recognition method, device, electronic device and storage medium to improve the above-mentioned problems.
[0004] According to one aspect of an embodiment of the present application, there is provided an object recognition method, comprising: acquiring a target image to be recognized; determining a recognition task corresponding to the target image, wherein the recognition task is used to indicate an object to be recognized; performing feature extraction on the target image by a first neural network to obtain a first feature map of the target image; performing object recognition on the first feature map by a target second neural network to obtain a recognition result of the target image corresponding to the recognition task, wherein the target second neural network refers to a second neural network corresponding to the size of the first feature map and the recognition task.
[0005] According to one aspect of an embodiment of the present application, there is provided an object recognition device, comprising: an acquisition module, for acquiring a target image to be recognized; a determination module, for determining a recognition task corresponding to the target image, wherein the recognition task is used to indicate an object to be recognized; a first feature extraction module, for performing feature extraction on the target image by a first neural network to obtain a first feature map of the target image; and a first recognition module, for performing object recognition on the first feature map by a target second neural network to obtain a recognition result of the target image corresponding to the recognition task, wherein the target second neural network refers to a second neural network corresponding to the size of the first feature map and the recognition task.
[0006] According to one aspect of an embodiment of the present application, an electronic device is provided, including: a processor; a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the object recognition method as described above is implemented.
[0007] According to one aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor, the object recognition method as described above is implemented.
[0008] In the scheme of the present application, the neural network model for object recognition is split, that is, the neural network model is split into a first neural network that has no requirements for the size of the input and has no correlation with the recognition task, and a second neural network that has size requirements for the input and is related to the recognition task, wherein the first neural network is used to extract features from the input image, and the second neural network is used to perform object recognition based on the feature map of the image and output the recognition result. On this basis, for the target image to be recognized, it is input into the first neural network for feature extraction to obtain the first feature map; and the corresponding target second neural network is determined according to the recognition task of the target image and the size of the first feature map, and then the target second neural network performs object recognition according to the first feature map to obtain the recognition result of the target image corresponding to the recognition task.
[0009] In the scheme of the present application, the first neural network has no correlation with the recognition task and has no requirements for the input size, so it can be applicable to images of different recognition tasks and images of various sizes; the part with input size requirements and related to the recognition task is used as the second neural network. Therefore, in the scheme of the application, for images of different sizes and different recognition tasks, only a relatively adaptive second neural network needs to be pre-built, and images of different sizes and different recognition tasks can reuse the first neural network. Compared with the related art, it is necessary to build an independent neural network model for object recognition for each size and each recognition task. For the neural network model, if the model is only applicable to one size and one recognition task, the utilization rate of computing resources will be reduced and the training time of the model will be increased.
[0010] The solution of the present application realizes the separation of the neural network for feature extraction and the neural network for object recognition, and only associates the neural network for object recognition with the recognition task, and has requirements on the input size, so that the neural network for feature extraction can be applicable to images of different sizes and different recognition tasks, thereby reducing the number of neural network models to be constructed and trained, reducing the time for training neural network models, while improving the utilization rate of computing resources and increasing the scope of application of neural network models.
[0011] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] The drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present application, and together with the specification are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0013] Figure 1 is a flowchart of an object recognition method according to an embodiment of the present application.
[0014] Figure 2 It is a flowchart of specific steps of step 120 according to an embodiment of the present application.
[0015] Figure 3 It is a flowchart of specific steps of step 140 according to an embodiment of the present application.
[0016] Figure 4 It is a schematic diagram showing the display of the detection results of the accessible space around the vehicle according to an embodiment of the present application.
[0017] Figure 5 It is a flowchart showing object recognition for a target image according to another embodiment of the application.
[0018] Figure 6 It is a flowchart of specific steps before step 140 according to an embodiment of the present application.
[0019] Figure 7 It is a flowchart of specific steps after step 630 according to an embodiment of the present application.
[0020] Figure 8 The flowchart of performing object recognition on a target image according to another embodiment of the present application is shown.
[0021] Fig. 9 The flowchart of performing object recognition on a target image according to another embodiment of the present application is shown.
[0022] Fig.10 It is a block diagram of an object recognition device according to an embodiment of the present application.
[0023] Fig.11 It is a hardware structure diagram of an electronic device according to an exemplary embodiment of the present application.
[0024] The above-mentioned drawings have shown clear embodiments of the present invention, which will be described in more detail hereinafter. These drawings and textual descriptions are not intended to limit the scope of the present invention in any way, but to illustrate the concept of the present invention to computer technicians in this field through specific embodiments. DETAILED DESCRIPTION
[0025] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this application will be more comprehensive and complete and fully convey the concept of the example embodiments to those skilled in the art.
[0026] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0027] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present application. However, those skilled in the art will appreciate that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, devices, steps, etc. can be adopted. In other cases, known methods, devices, realizations or operations are not shown or described in detail to avoid blurring the various aspects of the present application.
[0028] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices. The flowcharts shown in the accompanying drawings are merely exemplary and do not necessarily include all the contents and operations / steps, nor do they necessarily have to be executed in the order described. For example, some operations / steps may be further decomposed, while some operations / steps may be combined or partially combined, so the actual execution order may change according to the actual situation.
[0029] In the perception task of autonomous driving, multiple cameras are often needed to identify the environment and objects around the vehicle. In the existing technology, due to the different pixel imaging effects of different cameras and the limitations of technology, images collected by one camera can only be recognized using a fixed neural network, which makes the utilization rate of computing resources low.
[0030] Therefore, in order to overcome the above-mentioned defects, the present application provides an object recognition method, by sharing the first neural network used to extract shallow features to obtain a first feature map, and then determining the target second neural network for object recognition in the model set according to the recognition task of each image and the size of the first feature map, so that different target images to be identified corresponding to different recognition tasks can all use the first neural network to extract the shallow features of each target image to be identified, thereby improving the utilization rate of computing resources.
[0031] Figure 1 1 is a flow chart of an object recognition method according to an embodiment of the present application. The method can be executed by an electronic device with processing capabilities, such as a server, a cloud server, a vehicle terminal, etc., which is not specifically limited here. Figure 1 As shown, the method specifically comprises the following steps:
[0032] Step 110: Acquire a target image to be identified.
[0033] The target image refers to the image to be identified, and the target image is obtained by image acquisition by an image acquisition device. For example, if the image acquisition device is installed on a vehicle, the target image may be an image acquired by the image acquisition device on the vehicle. In a specific embodiment, a plurality of image acquisition devices may be installed on the vehicle, such as a left front camera for acquiring images in front of the left side of the vehicle, a right front camera for acquiring images in front of the right side of the vehicle, a front camera for acquiring images directly in front of the vehicle, a rear camera for acquiring images directly behind the vehicle, etc. The image to be identified acquired by any camera on the vehicle may be used as the target image in the present application.
[0034] Step 120: determine a recognition task corresponding to the target image, where the recognition task is used to indicate an object to be recognized.
[0035] The recognition task can be set according to the actual application scenario. For example, in the field of intelligent driving, the recognition task can be the task of identifying pedestrians, identifying lane lines, identifying fixed obstacles, identifying vehicles, identifying parking space lines, identifying traffic lights, and identifying the accessible space of vehicles (i.e., the location to which the vehicle can drive, driving space, DS). The recognition task indicates the object to be recognized. For example, the task of identifying pedestrians corresponds to the object to be recognized as pedestrians, and the task of identifying lane lines corresponds to the object to be recognized as lane lines.
[0036] In some embodiments, before step 120, the target image may be marked with a recognition task, for example, by marking a recognition task identifier, where a recognition task identifier is used to indicate a recognition task. After acquiring the target image, the recognition task corresponding to the target image may be determined based on the recognition task identifier marked for the target image.
[0037] In other embodiments, a recognition task corresponding to each image acquisition device can be set, so that after acquiring a target image, the image acquisition device from which the target image originated is determined, and the recognition task corresponding to the image acquisition device from which the target image originated is used as the recognition task corresponding to the target image.
[0038] In other embodiments, Figure 2 As shown, step 120 includes:
[0039] Step 210: Obtain installation location information of the image acquisition device from which the target image originates.
[0040] The image acquisition device may be a vehicle-mounted camera, a still camera, or other equipment with integrated image acquisition function (such as a driving recorder).
[0041] The installation location information is used to indicate the installation location of the corresponding image acquisition device. For example, on a vehicle, the image acquisition device may be a vehicle-mounted camera. The vehicle-mounted cameras include but are not limited to: a front-view camera, a side-view camera, and a rear-view camera. The corresponding installation location information of each vehicle-mounted camera is used to indicate the installation location on the vehicle.
[0042] Step 220 , based on the correspondence between the installation position and the recognition task, the recognition task corresponding to the installation position indicated by the installation position information is determined as the recognition task corresponding to the target image.
[0043] The correspondence between the installation position and the recognition task can be pre-set, wherein one installation position can correspond to one recognition task or to multiple recognition tasks, which is not specifically limited here and can be set according to actual needs.
[0044] Specifically, the correspondence between the installation position and the recognition task can be: the image acquisition device installed in front of the vehicle, its recognition task can be to recognize traffic lights in the target image, to recognize obstacles in the target image, etc.; the image acquisition device installed on the left side of the vehicle, its recognition task can be to recognize lane lines around the vehicle; the specific correspondence between the installation position and the recognition task can be set according to actual needs and is not specifically limited here.
[0045] In other embodiments, image acquisition devices at different installation locations may be pre-labeled differently, and a mapping relationship may be established between each label and a corresponding recognition task. After the installation location information of the image acquisition device from which the target image originates is determined, the recognition task corresponding to the image acquired by the image acquisition device may be determined based on the label of the image acquisition device.
[0046] Please continue reading Figure 1 , step 130, the first neural network performs feature extraction on the target image to obtain a first feature map of the target image.
[0047] The first feature map refers to a feature map output after the first neural network extracts features of the target image.
[0048] In some embodiments, the first neural network may include a convolutional neural network, a deep residual network, a deep residual shrinkage network, a forward feedback network, a Hopfield network, an inverse image network, an AlexNet network, a vgg network, a googlenet network, a resnet network, or one or more networks for extracting features of a target image. Specifically, in an embodiment of the present application, a resnet network is used to extract features of a target image. The resnet network has a pure convolutional network structure, which is characterized by having no requirements for input size. For example, the input of the first neural network is a H*W (where H is the length of the image and W is the width of the image) image, and the output is a C*H*W feature map, where C is the number of channels (i.e., the depth of the image).
[0049] Specifically, feature extraction of the target image may be performed on shallow features of the target image, for example, low-level visual features of the image, including texture, color, shape, etc. In a specific embodiment of the present application, the first neural network may be used to extract feature points, feature lines, and object contours in the target image.
[0050] In some embodiments, since the target image captured by the image acquisition device may have problems such as exposure and noise, resulting in poor quality of the target image, the target image may be preprocessed in advance to ensure the accuracy of the recognition result. Preprocessing the target image may include denoising the target image, white balancing the target image, and intercepting the target image. Compared with the target image before preprocessing, the target image after preprocessing has less noise, a more appropriate white balance ratio, higher clarity, and higher image quality.
[0051] Step 140, the target second neural network performs object recognition on the first feature map to obtain a recognition result of the target image corresponding to the recognition task, wherein the target second neural network refers to a second neural network corresponding to the size of the first feature map and the recognition task.
[0052] In the solution of the present application, multiple second neural networks can be pre-set according to the recognition task and the input size requirements, and the second neural networks can be trained to facilitate subsequent online applications.
[0053] After determining the recognition task corresponding to the target image and the size of the first feature map, the corresponding second neural network, namely the target second neural network, can be determined in combination with the recognition task and the size of the first feature map.
[0054] In some embodiments, the second neural network may be a fully connected network for classification, and the fully connected network may include one or more fully connected layers.
[0055] In other embodiments, the second neural network may include a cascaded second feature extraction network and a fully connected network, wherein the second feature extraction network is used to extract deep features (or deep features) from the first feature map output by the first neural network, and the second feature extraction network may be a convolutional neural network, a recurrent neural network, a long short-term memory neural network, etc.; the fully connected network is used to classify the feature map output by the second feature extraction network and output the classification recognition result.
[0056] For a fully connected network, different fully connected networks have different requirements for the input size. A fully connected network with one structure is adapted to an input image of one size. Therefore, for the second neural network including the fully connected network, it also has size requirements for the input, that is, the second neural network has a required input size, and the required input size is used to indicate the size of the input required by the second neural network, wherein the required input size corresponding to the second neural network is determined by the structure of the fully connected network in the second neural network.
[0057] The recognition result can be used to indicate whether the object indicated by the corresponding recognition task exists in the target image. If it is determined that it exists, the recognition result can further indicate the position of the object indicated by the recognition task in the target image. In a specific embodiment, if it is determined that the object indicated by the recognition task exists in the target image, the recognized object can be marked and displayed in the target image, for example, by box selection.
[0058] Figure 3 1 is a flowchart of specific steps of step 140 according to an embodiment of the present application. In some embodiments, the target second neural network includes a cascaded target convolutional neural network and a fully connected layer; step 140 includes:
[0059] Step 310: The target convolutional neural network performs deep feature extraction on the first feature map to obtain a second feature map.
[0060] The target convolutional neural network refers to the convolutional neural network in the target second neural network. Specifically, the target convolutional neural network in the target second neural network can have multiple layers, which is not limited here. In this embodiment, although the target convolutional neural network and the first neural network are both used for feature extraction, the first neural network is used to extract shallow features, which is equivalent to the extraction of coarse-grained features, and the target convolutional neural network is used to extract deep features, which is a fine-grained feature extraction. By performing deep feature extraction through the target convolutional neural network, the deep semantic information of the target image can be mined, thereby ensuring the accuracy of object recognition.
[0061] Step 320: The fully connected layer classifies the second feature map to obtain a recognition result of the target image corresponding to the recognition task.
[0062] In some specific embodiments, the second neural network can be set to N categories, and the parameters of the last fully connected layer are also set to N (that is, the number of neurons). In this way, the fully connected layer can output a 1*N multidimensional vector, where each dimension corresponds to a category. After passing through the softmax activation function, the numbers in each dimension of the multidimensional vector can be normalized to the range of 0 to 1. The data of each dimension can represent the probability that the second feature map is the corresponding category, and then the category with the highest probability value is output. It can be understood that the output result is the recognition result of the target image corresponding to the recognition task.
[0063] In the field of intelligent driving, the recognition results can be sent to the vehicle, which will display the recognition results. Furthermore, the vehicle can make driving decisions based on the recognition results, such as determining whether to adjust the driving direction, driving speed, or whether to stop.
[0064] Figure 4It is a schematic diagram of the result of identifying the accessible space of the vehicle according to an embodiment of the present application. Figure 4 The black dotted line in the middle shows the boundary line of the vehicle's accessible space (which can also be understood as the boundary line of the obstacle).
[0065] Assuming that the target image is an image with a size of H*Wmm, where H is the length of the target image and W is the width of the target image, the first neural network first performs shallow feature extraction on the image, such as feature lines. The first neural network outputs a first feature map with a size of H*Wmm for the target image, and then determines a target second neural network corresponding to the recognition task of the target image and the size of the first feature map. The target second neural network identifies the reachable space of the vehicle according to the first feature map corresponding to the target image. The target convolutional neural network in the target second neural network extracts the features of the boundary line based on the first feature map of the target image to obtain a second feature map; then the fully connected layer performs classification according to the second feature map and outputs a recognition result, which can indicate the coordinates of W points on the boundary line of the reachable space of the vehicle in the target image.
[0066] Furthermore, the server can send the coordinates of W points to the vehicle side, and the vehicle side marks the target image according to the coordinates of the W points and displays them on the screen of the vehicle side. Optionally, the server can also send the coordinates of the W points to an electronic device with a display function that is connected to the server for communication, such as a smart phone, a smart watch, a tablet computer, etc., which is not specifically limited here; the communication connection can be the same local area network, Bluetooth, ZigBee network, etc., which is not specifically limited here.
[0067] Figure 5 is a flow chart of an object recognition method according to another embodiment of the application. Figure 5 As shown, the target image to be identified includes a first target image 510 with a size of 400*300 mm and a second target image 520 with a size of 800*600 mm. The backbone (backbone network, i.e., the first neural network) 530 extracts features from the first target image 510 and the second target image 520 respectively, and outputs the corresponding first feature map. In this embodiment, the backbone is a Resnet network.
[0068] The backbone has no requirement for the size of the input target image, and the size of the input target image is the same as the size of the output feature map. Therefore, the backbone can output a first feature map with a size of 400*300mm for the first target image 510, and a first feature map with a size of 800*600mm for the second target image 520.
[0069] Then, the recognition task corresponding to the first target image 510 and the recognition task corresponding to the second target image 520 are determined according to the installation information of the image acquisition device from which the first target image 510 and the second target image 520 are obtained.
[0070] In this embodiment, it is assumed that a second neural network (for example, Figure 5 head α540 in ) and a second neural network for recognizing object II (e.g. Figure 5 The input size of headα540 is 400*300mm, and the input size of headβ550 is 800*600mm.
[0071] If the recognition task of the first target image 510 includes a task for recognizing object I (for ease of description, it is assumed to be the first recognition task), and the first feature map of the first target image 510 meets the input size requirement of headα540, then the first feature map of the first target image 510 can be input into headα540, and headα540 outputs a recognition result corresponding to the first recognition task based on the first feature map of the first target image 510.
[0072] If the recognition task of the second target image 520 includes a task for identifying object II (for ease of description, it is assumed to be the second recognition task), and the first feature map of the second target image 520 meets the input size requirement of headβ550, then the first feature map of the second target image 520 can be input into headβ550, and headβ550 outputs a recognition result corresponding to the second recognition task based on the first feature map of the second target image 520.
[0073] In the scheme of the present application, the neural network model for object recognition is split, that is, the neural network model is split into a first neural network that has no requirements for the size of the input and has no correlation with the recognition task, and a second neural network that has size requirements for the input and is related to the recognition task, wherein the first neural network is used to extract features from the input image, and the second neural network is used to perform object recognition based on the feature map of the image and output the recognition result. On this basis, for the target image to be recognized, it is input into the first neural network for feature extraction to obtain the first feature map; and the corresponding target second neural network is determined according to the recognition task of the target image and the size of the first feature map, and then the target second neural network performs object recognition according to the first feature map to obtain the recognition result of the target image corresponding to the recognition task.
[0074] In the scheme of the present application, the first neural network has no correlation with the recognition task and has no requirements for the input size, so it can be applicable to images of different recognition tasks and images of various sizes; the part with input size requirements and related to the recognition task is used as the second neural network. Therefore, in the scheme of the application, for images of different sizes and different recognition tasks, only a relatively adaptive second neural network needs to be pre-built, and images of different sizes and different recognition tasks can reuse the first neural network. Compared with the related art, it is necessary to build an independent neural network model for object recognition for each size and each recognition task. For the neural network model, if the model is only applicable to one size and one recognition task, the utilization rate of computing resources will be reduced and the training time of the model will be increased.
[0075] The solution of the present application realizes the separation of the neural network for feature extraction and the neural network for object recognition, and only associates the neural network for object recognition with the recognition task, and has requirements on the input size, so that the neural network for feature extraction can be applicable to images of different sizes and different recognition tasks, thereby reducing the number of neural network models to be constructed and trained, reducing the time for training neural network models, while improving the utilization rate of computing resources and increasing the scope of application of neural network models.
[0076] In some embodiments, Figure 6 As shown, before step 140, the method further includes:
[0077] Step 610, obtaining the recognition object information corresponding to each second neural network in the model set, where the recognition object information is used to indicate the object that the corresponding second neural network is used to recognize.
[0078] The recognition object information is used to indicate the object that the corresponding second neural network is used to recognize, that is, the recognition object information can be the main features that characterize the object that the second neural network is used to recognize. For example, when the recognition object is a traffic light, the recognition object information can be the color, shape and position information of the traffic light in the target image, etc.
[0079] In other embodiments, different identifiers may be set for different recognition objects, and the recognition object information may be an identifier set for the recognition object. When the identifiers set for the recognition objects corresponding to the second neural networks in the model set are obtained, it can be determined that the identifier corresponds to the object that the second neural network is used to recognize.
[0080] Step 620, matching the recognition task with the recognition object information corresponding to each second neural network, and determining in the model set a candidate second neural network in which the object indicated by the corresponding recognition object information matches the object indicated by the recognition task.
[0081] In some embodiments, when an object indicated by a recognition task is the same as an object indicated by recognition object information corresponding to a second neural network in a model set, it can be determined that the object indicated by the recognition task matches the object indicated by the recognition object information corresponding to the second neural network, thereby determining that the second neural network is a candidate second neural network.
[0082] In other embodiments, when the object indicated by the recognition object information corresponding to a second neural network includes the object indicated by a recognition task, it can be determined that the object indicated by the recognition task matches the object indicated by the recognition object information corresponding to the second neural network.
[0083] Step 630, matching the size of the first feature map with the required input size corresponding to each candidate second neural network, determining a candidate second neural network whose corresponding required input size is the same as the size of the first feature map; and using the determined candidate second neural network whose required input size is the same as the size of the first feature map as the target second neural network.
[0084] In this embodiment, the recognition task is matched with the recognition object information corresponding to each second neural network to confirm the candidate second neural network whose recognition object indicated by the recognition object information in the second neural network is the same as the recognition object indicated by the recognition task, and then the target second neural network whose input size is required to be the same as the size of the first feature map in the candidate second neural network is determined, so that the target second neural network can recognize the first feature map.
[0085] In some embodiments, after step 630, the method further includes:
[0086] Step 710: If it is determined that there is no candidate second neural network whose corresponding required input size is the same as the size of the first feature map, a candidate second neural network is selected as the target second neural network.
[0087] Step 720, resize the first feature map according to the required input size corresponding to the target second neural network, so that the size of the adjusted first feature map is the same as the required input size corresponding to the target second neural network.
[0088] In a specific embodiment, in order to enable the target second neural network to recognize the first feature map, the size of the first feature map can be adjusted according to the interpolation algorithm. The adjustment can be enlargement or reduction, depending on the specific situation. The interpolation algorithm can be used to enlarge or reduce the image. The interpolation algorithm is often used in the image field to adjust the image size. The points in the new image matrix are calculated from the points in the old image matrix and inserted to generate images of different sizes.
[0089] Optionally, the nearest neighbor interpolation algorithm, bilinear interpolation algorithm, and bicubic interpolation algorithm can be used. In a specific embodiment, the bilinear interpolation algorithm is used to resize the first feature map. The bilinear interpolation algorithm is also called the first-order interpolation method. The grayscale value of the sampling point is first calculated based on the coordinates and grayscale values of the four points around the sampling point (i.e., a point in the old image matrix mentioned above). First, the coordinates of the four points in the horizontal x direction are firstly subjected to first-order linear interpolation (two first-order linear interpolations are required), and then the coordinates of the four points in the vertical y direction are subjected to first-order linear interpolation (only one first-order linear interpolation is required). For example, the coordinates of a sampling point , the coordinates of the four points around point P are , , , , first calculate the first-order linear interpolation of point P in the x direction and get two temporary points and The grayscale value of point P in the x direction is calculated as follows:
[0090] (Formula 1)
[0091] (Formula 2)
[0092] in, Indicates temporary point Gray value in the x direction, Indicate point The gray value of Indicates point Gray value, Indicates temporary point Gray value in the x direction, Indicate point The gray value of Indicates point Grayscale value.
[0093] Then calculate the first-order linear interpolation of point P in the y direction. The specific calculation of the first-order linear interpolation of point P in the y direction is
[0094] The method is as follows:
[0095] (Formula 3)
[0096] in, Represents the grayscale value of point P in the y direction.
[0097] Then substitute formula (1) and formula (2) into formula (3) to obtain the grayscale value of point P. Specifically, the grayscale value of point P is as follows:
[0098]
[0099] Then, the new coordinates of point P are determined according to the coordinates of point P in the original image, the size of the original image, and the required size. The specific calculation can be based on the following formula:
[0100] (Formula 4)
[0101] (Formula 5)
[0102] Where (dstX, dsY) is the coordinate of point P in the original image, srcWidth is the width of the original image, dstWidth is the width of the new image, srcHeight is the length of the original image, dstHeight is the length of the new image, and (srcX, srcY) is the coordinate of point P in the new image. When using the above formulas (4) and (5), the geometric center of the original image does not coincide with the geometric center of the new image, and the pixel points near the boundary line of the original image cannot be bilinearly interpolated. In order to solve these problems, the above formulas (4) and (5) are optimized as follows:
[0103] (Formula 6)
[0104] (Formula 7)
[0105] The grayscale value of each pixel in the new image and the coordinates of each pixel in the new image are calculated according to the above formulas, so as to enlarge or reduce the original image.
[0106] Step 730: The target second neural network performs object recognition on the adjusted first feature map to obtain a recognition result of the target object corresponding to the recognition task.
[0107] The size of the adjusted first feature map meets the required input size of the target second neural network, and the adjusted first feature map is input into the target second neural network, so that the target neural network can perform recognition based on the first feature map.
[0108] In this embodiment, when there is no candidate second neural network whose corresponding required input size is the same as the size of the first feature map, a candidate second neural network is selected as the target second neural network, and then the size of the first feature map is adjusted according to the required input size corresponding to the target second neural network, so that the size of the first feature map is the same as the required input size corresponding to the target second neural network, and the adjusted first feature map is recognized to obtain the recognition result of the target image. In this way, the scope of use of the neural network model is increased; during the training process, images collected by image acquisition devices at multiple locations can also be combined for training.
[0109] In order to ensure the accuracy of the recognition result, the first neural network and the second neural network need to be trained in advance. Specifically, a sample set is pre-constructed, and the sample set includes multiple sample images and label information of the sample images, wherein the label information of the sample image is used to indicate the position information of the object indicated by the recognition task corresponding to the sample image in the sample image. The sample set includes sample images of various sizes and sample images corresponding to various recognition tasks.
[0110] During the training process, each sample image is input into the first neural network respectively, and shallow features are extracted to obtain the first feature map of each sample image. Then, the target second neural network corresponding to the size of the first feature map of the sample image and the recognition task corresponding to the sample image is determined. The target second neural network outputs the sample recognition result corresponding to the recognition task corresponding to the sample image based on the first feature map of the sample image. It can be understood that the sample recognition result indicates whether the sample image includes the object indicated by the recognition task corresponding to the sample image, and the position information of the object indicated by the recognition task corresponding to the sample image in the sample image.
[0111] Then, the loss value of the loss function is calculated based on the label information of the sample image and the sample recognition result corresponding to the sample image. If the loss value does not converge, the parameters of the first neural network and the target second neural network are adjusted in reverse, and the sample recognition result is output again for the sample image through the first neural network and the target second neural network after adjusting the parameters, and the loss value of the loss function is calculated again until the loss value converges.
[0112] The above process is repeated for each sample image, and when the training end condition is reached, the training of the first neural network and the second neural network is terminated. Afterwards, the first neural network and the second neural network are used for online object recognition to ensure the accuracy of object recognition.
[0113] Figure 8 is a flow chart of an object recognition method according to another embodiment of the application. Figure 8As shown, the target image to be identified includes a first target image 510 with a size of 400*300 mm and a second target image 520 with a size of 800*600 mm. Backbone α 810 (first neural network) extracts features from the first target image 510 and the second target image 520 respectively, and outputs corresponding first feature maps. In this embodiment, backbone α 810 is a Resnet network.
[0114] Backbone α 810 has no requirement for the size of the input target image, and the size of the input target image is the same as the size of the output feature map. Therefore, backbone α 810 can output a feature map with a size of 400*300 mm for the first target image 510, and a feature map with a size of 800*600 mm for the second target image 520.
[0115] Then, the recognition task corresponding to the first target image 510 and the recognition task corresponding to the second target image 520 are determined according to the installation information of the image acquisition device from which the first target image 510 and the second target image 520 are obtained.
[0116] In this embodiment, it is assumed that a second neural network for recognizing object I and a second neural network for recognizing object II are provided, wherein the input requirement size of headα540 is 400*300 mm, and the input requirement size of headβ550 is 800*600 mm.
[0117] If the recognition task of the first target image 510 includes a task for recognizing object I (for ease of description, it is assumed to be the first recognition task), and the first feature map of the first target image 510 meets the input size requirement of headα540, then the first feature map of the first target image 510 can be directly input into headα540, and headα540 outputs a recognition result corresponding to the first recognition task based on the first feature map of the first target image 510.
[0118] If the recognition task of the first target image includes a task for recognizing object II (for ease of description, it is assumed to be the second recognition task), but the first feature map of the first target image 510 does not meet the input size requirement of headβ550, the size of the first feature map of the first target image 510 is first adjusted by resize 820 so that the size of the adjusted first feature map of the first target image is the same as the required input size of headβ550 (i.e., 800*600mm), and then the adjusted first feature map of the first target image is input into headβ550, which outputs a recognition result corresponding to the second recognition task based on the first feature map of the first target image.
[0119] It can be understood that if the recognition task of the second target image 520 includes the first recognition task, but the first feature map of the second target image 520 does not meet the input size requirement of headα540, the size of the first feature map of the second target image 520 is first adjusted by resize 820 so that the size of the adjusted first feature map of the second target image is the same as the required input size of headα540 (i.e., 400*300mm), and then the adjusted first feature map of the second target image is input into headα540, and headα540 outputs the recognition result corresponding to the first recognition task based on the first feature map of the second target image.
[0120] If the recognition task of the second target image 520 includes a second recognition task, and the first feature map of the second target image 520 meets the input size requirement of headβ550, the first feature map of the second target image 520 can be directly input into headβ550, and headβ550 outputs a recognition result corresponding to the second recognition task based on the first feature map of the second target image 520.
[0121] In some embodiments, the size of the feature map output by the first neural network is the same as the size of the image input to the first neural network. In a specific embodiment, before step 130, the method further includes:
[0122] The target image is resized so that the size of the resized target image is the same as the required input size corresponding to the target second neural network.
[0123] Specifically, the first neural network can use a Resnet (residual neural network) network, which has a pure convolutional network structure inside. The characteristics of the Resnet network are that there is no requirement for the size of the input image, and the size of the input image is the same as the size of the output image. After the target image to be identified is extracted by the first neural network, the number of channels of the output first feature map changes, but the size of the image is the same as the size of the target image. For example, the input is a target image of H*Wmm (where H is the length of the target image and W is the width of the target image), and the first neural network outputs a first feature map of C*H*Wmm, where C is the number of channels.
[0124] In some specific implementations, the target image may be first adjusted according to the required input size corresponding to the target second neural network, and the adjusted target image may be input into the first neural network.
[0125] Fig. 9 is a flow chart of an object recognition method according to another embodiment of the application. Fig. 9As shown, the target image to be identified includes a first target image 510 with a size of 400*300 mm and a second target image 520 with a size of 800*600 mm, and the identification task corresponding to the first target image 510 and the identification task corresponding to the second target image 520 are determined according to the installation information of the image acquisition device from which the first target image 510 and the second target image 520 are derived.
[0126] In this embodiment, it is assumed that a second neural network (for example, Fig. 9 The input size of headα540 is required to be 400*300mm.
[0127] If the recognition task of the first target image includes a task for recognizing object I (for ease of description, it is assumed to be the first recognition task), and the first target image meets the input size requirement of headα540, then the first target image can be directly input into backboneβ910 for feature extraction to obtain the first feature map of the first target image. In this embodiment, backboneβ910 is a Resnet network. Backboneβ910 has no requirements for the size of the input target image, and the size of the input target image is the same as the size of the output feature map. Therefore, backboneβ910 can output a first feature map with a size of 400*300mm for the first target image 510, and then input the first feature map of the first target image 510 into headα540, and headα540 outputs the recognition result corresponding to the first recognition task based on the first feature map of the first target image 510.
[0128] If the recognition task of the second target image 520 includes the first recognition task, but the second target image 520 does not meet the input size requirement of headα540, the size of the second target image 520 is first adjusted by resize 820 so that the size of the adjusted second target image is the same as the required input size of headα540 (i.e., 400*300mm), and then the adjusted second target image is input into backboneβ910 for feature extraction to obtain the first feature map of the adjusted second target image. Backboneβ910 can output a first feature map with a size of 400*300mm for the adjusted second target image, and then the first feature map of the adjusted second target image is input into headα540, and headα540 outputs the recognition result corresponding to the first recognition task based on the first feature map of the adjusted second target image.
[0129] Fig.10 is a block diagram of an object recognition device according to an embodiment of the present application. Figure 8 As shown, the object recognition device 1000 includes: an acquisition module 1010 , a determination module 1020 , a first feature extraction module 1030 and a first recognition module 1040 .
[0130] An acquisition module 1010 is used to acquire a target image to be identified; a determination module 1020 is used to determine an identification task corresponding to the target image, wherein the identification task is used to indicate an object to be identified; a first feature extraction module 1030 is used to perform feature extraction on the target image by a first neural network to obtain a first feature map of the target image; a first identification module 1040 is used to perform object identification on the first feature map by a target second neural network to obtain an identification result of the target image corresponding to the identification task, wherein the target second neural network refers to a second neural network corresponding to the size of the first feature map and the identification task.
[0131] In some embodiments, the object recognition device 1000 also includes: a recognition object information acquisition module, used to obtain recognition object information corresponding to each second neural network in the model set, the recognition object information is used to indicate the object used to be recognized by the corresponding second neural network; a candidate second neural network determination module, used to match the recognition task with the recognition object information corresponding to each second neural network, and determine in the model set a candidate second neural network whose object indicated by the corresponding recognition object information matches the object indicated by the recognition task; a target second neural network determination module, used to match the size of the first feature map with the required input size corresponding to each candidate second neural network, and determine a candidate second neural network whose corresponding required input size is the same as the size of the first feature map; and use the determined candidate second neural network whose required input size is the same as the size of the first feature map as the target second neural network.
[0132] In some embodiments, the object recognition device 1000 also includes: a selection module, which is used to select a candidate second neural network as the target second neural network if it is determined that there is no candidate second neural network whose corresponding required input size is the same as the size of the first feature map; a first size adjustment module, which is used to resize the first feature map according to the required input size corresponding to the target second neural network, so that the size of the adjusted first feature map is the same as the required input size corresponding to the target second neural network; a second recognition module, which is used to perform object recognition on the adjusted first feature map by the target second neural network to obtain a recognition result of the target object corresponding to the recognition task.
[0133] In some embodiments, the size of the feature map output by the first neural network is the same as the size of the image input to the first neural network; the object recognition device 1000 also includes: a second size adjustment module, used to adjust the size of the target image so that the size of the adjusted target image is the same as the required input size corresponding to the target second neural network.
[0134] In some embodiments, the target image corresponds to at least two recognition tasks.
[0135] In some embodiments, the determination module 1020 includes: an installation position information acquisition unit, used to acquire the installation position information of the image acquisition device from which the target image originates; and a recognition task determination unit, used to determine the recognition task corresponding to the installation position indicated by the installation position information as the recognition task corresponding to the target image based on the correspondence between the installation position and the recognition task.
[0136] In some embodiments, the target second neural network includes a cascaded target convolutional neural network and a fully connected layer; the first recognition module 1040 includes: a feature extraction unit, which is used to perform deep feature extraction on the first feature map by the target convolutional neural network to obtain a second feature map; a classification unit, which is used to classify the second feature map by the fully connected layer to obtain a recognition result of the target image corresponding to the recognition task.
[0137] According to one aspect of an embodiment of the present application, an electronic device is also provided, which includes: a processor; a memory, wherein computer-readable instructions are stored in the memory, and when the computer-readable instructions are executed by the processor, the method in any of the above embodiments is implemented.
[0138] According to one aspect of the embodiments of the present application, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method in any of the above embodiments.
[0139] In one aspect of the embodiments of the present application, an electronic device is also provided, such as Fig.11 As shown, the electronic device 1100 includes a processor 1110 and one or more memories 1120. The one or more memories 1120 are used to store program instructions executed by the processor 1110. When the processor 1110 executes the program instructions, the above-mentioned object recognition method is implemented.
[0140] Furthermore, the processor 1110 may include one or more processing cores. The processor 1110 runs or executes instructions, programs, code sets or instruction sets stored in the memory 1120, and calls data stored in the memory 1120. Optionally, the processor 1110 may be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 1110 may integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. It is understandable that the above-mentioned modem may not be integrated into the processor, but may be implemented separately through a communication chip.
[0141] According to one aspect of the present application, the present application further provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments; or may exist independently without being assembled into the electronic device. The above computer-readable storage medium carries computer-readable instructions, and when the computer-readable storage instructions are executed by a processor, the method in any of the above embodiments is implemented.
[0142] It should be noted that the computer-readable medium shown in the embodiment of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.
[0143] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.
[0144] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.
[0145] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0146] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the embodiments disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application.
[0147] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.
Claims
1. An object recognition method, characterized in that: The method comprises: Acquire a target image to be identified; Determining a recognition task corresponding to the target image, where the recognition task is used to indicate an object to be recognized, wherein determining the recognition task corresponding to the target image includes: Acquire installation location information of an image acquisition device from which the target image originates; Based on the correspondence between the installation position and the recognition task, determining the recognition task corresponding to the installation position indicated by the installation position information as the recognition task corresponding to the target image; Performing feature extraction on the target image by a first neural network to obtain a first feature map of the target image; Acquire identification object information corresponding to each second neural network in the model set, wherein the identification object information is used to indicate an object used to be identified by the corresponding second neural network; Matching the recognition task with the recognition object information corresponding to each second neural network, and determining in the model set a candidate second neural network whose object indicated by the corresponding recognition object information matches the object indicated by the recognition task; Matching the size of the first feature map with the required input size corresponding to each of the candidate second neural networks, determining a candidate second neural network whose corresponding required input size is the same as the size of the first feature map; and using the determined candidate second neural network whose required input size is the same as the size of the first feature map as a target second neural network; The target second neural network performs object recognition on the first feature map to obtain a recognition result of the target image corresponding to the recognition task, wherein the target second neural network refers to a second neural network corresponding to the size of the first feature map and the recognition task.
2. The method according to claim 1, characterized in that After matching the recognition task with the recognition object information corresponding to each second neural network and determining in the model set a candidate second neural network whose object indicated by the corresponding recognition object information matches the object indicated by the recognition task, the method further includes: If it is determined that there is no candidate second neural network with the same required input size as the size of the first feature map, selecting a candidate second neural network as the target second neural network; According to the required input size corresponding to the target second neural network, resizing the first feature map so that the size of the adjusted first feature map is the same as the required input size corresponding to the target second neural network; The target second neural network performs object recognition on the adjusted first feature map to obtain a recognition result of the target image corresponding to the recognition task.
3. The method according to claim 1, characterized in that The size of the feature map output by the first neural network is the same as the size of the image input to the first neural network; Before extracting features from the target image by the first neural network to obtain a first feature map of the target image, the method further includes: The target image is resized so that the size of the resized target image is the same as the required input size corresponding to the target second neural network.
4. The method according to claim 1, characterized in that: The target image corresponds to at least two recognition tasks.
5. The method according to claim 1, characterized in that The target second neural network includes a cascaded target convolutional neural network and a fully connected layer; The step of performing object recognition on the first feature map by the target second neural network to obtain a recognition result of the target image corresponding to the recognition task includes: The target convolutional neural network performs deep feature extraction on the first feature map to obtain a second feature map; The second feature map is classified by the fully connected layer to obtain a recognition result of the target image corresponding to the recognition task.
6. An object recognition device, characterized in that: The device comprises: An acquisition module, used for acquiring a target image to be identified; A determination module is used to determine a recognition task corresponding to the target image, wherein the recognition task is used to indicate an object to be recognized, wherein the determination module includes: An installation location information acquisition unit, used to acquire the installation location information of the image acquisition device from which the target image originates; a recognition task determining unit, configured to determine, based on a correspondence between the installation position and the recognition task, a recognition task corresponding to the installation position indicated by the installation position information as a recognition task corresponding to the target image; A first feature extraction module, configured to extract features of the target image using a first neural network to obtain a first feature map of the target image; An identification object information acquisition module, used to acquire identification object information corresponding to each second neural network in the model set, wherein the identification object information is used to indicate an object used to be identified by the corresponding second neural network; a candidate second neural network determination module, configured to match the recognition task with the recognition object information corresponding to each second neural network, and determine in the model set a candidate second neural network whose object indicated by the corresponding recognition object information matches the object indicated by the recognition task; a target second neural network determination module, configured to match the size of the first feature map with the required input size corresponding to each of the candidate second neural networks, determine the candidate second neural network whose corresponding required input size is the same as the size of the first feature map; and use the determined candidate second neural network whose required input size is the same as the size of the first feature map as the target second neural network; The first recognition module is used to perform object recognition on the first feature map by a target second neural network to obtain a recognition result of the target image corresponding to the recognition task, wherein the target second neural network refers to a second neural network corresponding to the size of the first feature map and the recognition task.
7. An electronic device, characterized in that: The electronic device comprises: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 5 is implemented.
8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 5.
Citation Information
Patent Citations
Method and apparatus for generating image recognition model
CN110288049A
Multi-task detection method and device and storage medium
CN111046973A
