Identification method, device, equipment, and computer-readable storage medium

Abstract

The embodiments of the present application disclose a recognition method and apparatus, device, and computer-readable storage medium. The method includes: extracting an image frame to be recognized from a video stream; inputting the image frame to be recognized into a trained recognition model to obtain information about the target object and target scene information output by the recognition model; the recognition model includes an image feature extraction network for identifying and extracting image features of the image frame to be recognized, a detection task branch for outputting detection information of the object in the image frame, a semantic segmentation task branch for outputting semantic information of pixel points in the image frame, and a classification task branch for outputting scene type information in the image frame, determining information about the target object based on the detection information and semantic information, and determining target scene information based on the scene type information and semantic information. Multi-task processing is performed simultaneously for the same image feature, saving data processing time, and the results output by the task branches refer to each other, making the output information more accurate.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an identification method and apparatus, device, and computer-readable storage medium. Background Art

[0002] A video stream consists of multiple continuous image frames. The existing technology extracts features from the image frames to obtain image features, and then performs recognition and analysis on the image frames based on the extracted image features, such as identifying target objects in the image frames.

[0003] As the variety of image frame information required increases, the total amount of image frame information increases, leading to a longer timeframe for identifying and acquiring image features within the image frames. In particular, existing technologies cannot simultaneously identify both the target object and the target scene within an image frame, preventing increased recognition time. Furthermore, the simultaneous execution of multiple recognition steps hinders the accuracy of the recognition results.

[0004] Therefore, there is an urgent need for a recognition method that can simultaneously identify target object information and target scene information in an image frame to solve the above problems and ensure the accuracy of the recognition information without increasing the delay. Summary of the Invention

[0005] To solve the above technical problems, the embodiments of the present application respectively provide an identification method and apparatus, a device, and a computer-readable storage medium, which can simultaneously identify the target object information and the target scene information in the image frame without increasing the delay.

[0006] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by practice of the present application.

[0007] According to one aspect of an embodiment of the present application, a recognition method is provided, comprising: extracting an image frame to be recognized from a video stream; inputting the image frame to be recognized into a trained recognition model to obtain target object information and target scene information output by the recognition model; wherein the recognition model comprises an image feature extraction network for identifying and extracting image features of the image frame to be recognized, a detection task branch for outputting detection information of the object in the image frame, a semantic segmentation task branch for outputting semantic information of pixel points in the image frame, and a classification task branch for outputting scene type information in the image frame, determining the target object information based on the detection information and the semantic information, and determining the target scene information based on the scene type information and the semantic information.

[0008] Furthermore, the recognition method also includes: constructing an initial recognition model, the initial recognition model includes an image feature extraction network, the detection task branch, the semantic segmentation task branch and the classification task branch; inputting the image frame to be trained and recognized into the initial recognition model, the image feature extraction network recognizes and extracts features of the image frame to be trained and obtains the image features to be trained; the detection task branch outputs the first detection information of the object in the image frame corresponding to the image feature to be trained, the semantic segmentation task branch outputs the first semantic information of the pixel point in the image frame corresponding to the image feature to be trained, and the classification task branch outputs the first scene type information in the image frame corresponding to the image feature; the initial recognition model is corrected according to the first detection information, the first semantic information and the first scene information to obtain the trained recognition model.

[0009] Furthermore, the initial recognition model is corrected according to the first detection information, the first semantic information, and the first scene information to obtain the trained recognition model, including: determining a detection information loss function value according to the first detection information and the first standard detection information; determining a semantic information loss function value according to the first semantic information and the first semantic standard information; determining a scene type information loss function value according to the first scene type information and the first standard scene type information; and correcting the initial recognition model based on the detection information loss function value, the semantic information loss function value, and the scene type information loss function value to obtain the trained recognition model.

[0010] Furthermore, the initial recognition model is corrected based on the detection information loss function value, the semantic information loss function value, and the scene type information loss function value to obtain the trained recognition model, including: calculating a first back propagation value based on the detection information loss function value and a first dynamic modulation factor corresponding to the detection task branch; calculating a second back propagation value based on the semantic information loss function value and a second dynamic modulation factor corresponding to the semantic segmentation task branch; calculating a third back propagation value based on the scene type information loss function value and a third dynamic modulation factor corresponding to the classification task branch; and updating the configuration parameters in the initial recognition model according to the first back propagation value, the second back propagation value, and the third back propagation value to obtain the trained recognition model.

[0011] Furthermore, the detection information includes the position of the detection regression box in the image frame and the category of the predicted object, the semantic information includes the number and position of pixels corresponding to the predicted object in the image frame, and the semantic type corresponding to the pixels in the image frame, the information of the target object includes the category and position of the target object, and determining the information of the target object based on the detection information and the semantic information includes: determining the number of pixels corresponding to the predicted object in the detection regression box according to the position of the detection regression box; if the number of pixels corresponding to the predicted object is greater than the number of pixels of a preset category object corresponding to the category of the predicted object, then determining the position of the detection regression box as the position of the target object, and determining the category of the predicted object as the category of the target object.

[0012] Furthermore, the semantic type includes: scene type and object type; the scene type information includes the scene type in the image frame and the scene probability value corresponding to the scene type, and the target scene information is determined based on the scene type information and the semantic information, including: if the scene type in the semantic type is the same as the scene type in the scene type information, then determining that the scene type in the scene type information is the scene type of the target scene; if the scene type in the semantic type is different from the scene type in the scene type information, and the scene probability value corresponding to the scene type is greater than a preset scene probability threshold, then determining that the scene type in the scene type information is the scene type of the target scene; if the scene type in the semantic type is different from the scene type in the scene type information, and the scene probability value corresponding to the scene type is less than or equal to the preset scene probability threshold, then determining that the scene type in the semantic type corresponding to the pixel point in the image feature is the scene type of the target scene.

[0013] Furthermore, extracting the image frame to be identified from the video stream includes: obtaining a video stream, the video stream including multiple image frames; detecting whether the target object exists in each image frame, and determining the image frame in which the target object is detected as the image frame to be identified. Furthermore, before obtaining the trained recognition model, the method also includes: correcting the initial recognition model to obtain a corrected recognition model; and performing quantitative perception training of the INT8 edge computing module on the corrected recognition model to obtain a trained recognition model.

[0014] Furthermore, the configuration parameters in the initial recognition model are updated according to the first back propagation value, the second back propagation value, and the third back propagation value to obtain the trained recognition model, including: training the parameters in the detection task branch according to the first back propagation value to obtain a trained detection task branch; training the parameters in the semantic segmentation task branch according to the second back propagation value to obtain a trained semantic segmentation task branch; training the parameters in the classification task branch according to the third back propagation value to obtain a trained classification task branch; and obtaining the trained recognition model.

[0015] According to one aspect of an embodiment of the present application, a recognition device is provided, comprising: an extraction module configured to extract image features of an image frame to be recognized; an output module configured to input the image features into a trained recognition model to obtain target object information and target scene information output by the recognition model; wherein the recognition model comprises a detection task branch for outputting detection information of the object in the image features, a semantic segmentation task branch for outputting semantic information of pixel points in the image features, and a classification task branch for outputting scene type information in the image features, the target object information is determined based on the detection information and the semantic information, and the target scene information is determined based on the scene type information and the semantic information.

[0016] According to one aspect of an embodiment of the present application, an electronic device is provided, comprising: a controller; and a memory for storing one or more programs, wherein when the one or more programs are executed by the controller, the controller implements the above-described identification method.

[0017] According to one aspect of an embodiment of the present application, a computer-readable storage medium is further provided, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor of a computer, the computer executes the above-mentioned identification method.

[0018] According to one aspect of an embodiment of the present application, a computer program product or computer program is also provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the above-described identification method.

[0019] In the technical solution provided in the embodiments of the present application, the image features of the image frame to be identified are extracted, and the image features are input into a trained recognition model to obtain the target object information and target scene information output by the recognition model. The recognition model includes three task branches: a detection task branch for outputting detection information of the object in the image features, a semantic segmentation task branch for outputting semantic information of the pixel points in the image features, and a classification task branch for outputting scene type information in the image features. By performing multi-task processing on the same image feature at the same time, data processing time is saved. Finally, the target object information is determined based on the detection information and semantic information, and the target scene information is determined based on the scene type information and semantic information. The results output by the three task branches refer to each other, making the output related information more accurate.

[0020] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings are incorporated into and constitute a part of the specification, illustrating embodiments consistent with the present application and, together with the specification, serving to explain the principles of the present application. It is obvious that the drawings described below are merely some embodiments of the present application, and a person of ordinary skill in the art can derive other drawings based on these drawings without inventive effort. In the drawings:

[0022] Figure 1 It is a schematic diagram of an implementation environment involved in this application;

[0023] Figure 2 is a flow chart of an intelligent road recognition method shown in an exemplary embodiment of the present application;

[0024] Figure 3 is a flow chart of an identification method shown in an exemplary embodiment of the present application;

[0025] Figure 4 is a flowchart showing the steps of constructing a recognition model according to another exemplary embodiment of the present application;

[0026] Figure 5 is a structural diagram of a recognition model shown in another exemplary embodiment of the present application;

[0027] Figure 6 is a flowchart of a process of correcting a recognition model based on a loss function, shown in another exemplary embodiment of the present application;

[0028] Figure 7 is a flowchart of a process of correcting a recognition model based on back-propagation values, shown in another exemplary embodiment of the present application;

[0029] Figure 8 is a schematic diagram of performing edge-end quantization perception training on a recognition model according to another exemplary embodiment of the present application;

[0030] Figure 9 is a schematic diagram of the structure of an identification device shown in an exemplary embodiment of the present application;

[0031] Figure 10 It is a structural diagram of a computer system of an electronic device shown in an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0032] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. When the following description refers to the drawings, identical numerals in different figures represent identical or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0033] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically separate entities. That is, these functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0034] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, while others may be combined or partially combined. Therefore, the actual execution order may vary depending on the actual situation.

[0035] In this application, "plurality" refers to two or more. "And / or" describes the relationship between related objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, and B exists alone. The character " / " generally indicates that the related objects are in an "or" relationship.

[0036] First, see Figure 1 , Figure 1 Schematic diagram of an implementation environment involved in this application. The implementation environment includes a collection terminal 100 and a server 200. The terminal 100 and the server 200 communicate with each other via a wired or wireless network.

[0037] The acquisition terminal 100 has the function of acquiring video streams and can transmit the acquired video streams to the server 200. The acquisition terminal 100 includes but is not limited to any electronic device that can realize image visualization, such as a video camera, a still camera, a video camera, a mobile phone, and a vehicle-mounted video device, and is not limited here.

[0038] The server 200 can extract the image frames to be identified in the video stream, and then input the image frames to be identified into the trained recognition model, so that the recognition model outputs the target object information and target scene information. The server 200 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, wherein multiple servers can form a blockchain, and the server is a node on the blockchain. The server 200 can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network, content distribution network) and big data and artificial intelligence platforms, and this is not limited here.

[0039] In some scenarios, the acquisition terminal 100 and the server 200 can be placed in the same physical device or equipment. For example, the recognition method of the present application is used in the scenario of intelligent road recognition. The acquisition terminal 100 is an on-board video device in a road-running vehicle, which can collect video streams of relevant roads, and the server 200 is located inside the vehicle, such as Figure 2 As shown, Figure 2 This is a flow chart illustrating an intelligent road recognition method according to an exemplary embodiment of the present application. In the figure, image frames to be recognized are extracted from a real-time onboard video stream and fed into a trained recognition model. The recognition model includes a detection task branch, a semantic segmentation task branch, and a classification task branch. The model outputs target obstacle information and road information from the image frame and controls the vehicle based on the relevant information. Target obstacle information includes its location, size, and type, while road information includes information such as road type. Road conditions change rapidly during vehicle operation. The recognition method described in the following embodiments of the present application can accurately identify target obstacles and road types in real time. Subsequently, vehicle operating parameters are controlled based on the identified relevant information. For example, when a target obstacle is identified on the road, the vehicle is controlled to stop within a preset reaction time or immediately. If the road type is identified as a sloped road, the vehicle gear can be automatically switched. For detailed steps of the recognition method executed by server 200, please refer to the description in the following embodiments.

[0040] See also Figure 3 , Figure 3 This is a flow chart of an identification method shown in an exemplary embodiment of the present application. The method can be Figure 1The server 200 on the base station side in the implementation environment shown is specifically executed. Of course, the method can also be applied to other implementation environments and executed by server devices in other implementation environments, and this embodiment is not limited to this. Figure 3 As shown, the method includes at least S310 to S320, which are described in detail as follows:

[0041] S310: Extracting image frames to be identified from the video stream.

[0042] The image frame to be identified comes from a video stream. The video stream in this embodiment is acquired by the acquisition terminal 100 , and the video stream consists of multiple image frames.

[0043] S320: Input the image features of the image frame to be identified into the trained recognition model to obtain the target object information and target scene information output by the recognition model; wherein the recognition model includes an image feature extraction network for identifying and extracting the image features of the image frame to be identified, a detection task branch for outputting detection information of the object in the image features, a semantic segmentation task branch for outputting semantic information of the pixel points in the image features, and a classification task branch for outputting scene type information in the image features, determining the target object information based on the detection information and the semantic information, and determining the target scene information based on the scene type information and the semantic information.

[0044] Image features are feature images or feature data obtained after identifying and extracting features from the image frame to be identified. This embodiment does not limit its specific representation.

[0045] In this embodiment, the target object is an object in the image frame to be identified. For example, in a road scene, the target object may be a target obstacle on the road. The target scene information is scene information in the image frame to be identified, including road type, road weather, etc.

[0046] S320 is exemplified. In a vehicle road scene, the extracted image frame to be identified is input into a trained recognition model, and the recognition model outputs information about the target obstacle in the image frame to be identified and information about the target road scene. The information about the target obstacle includes: obstacle type, obstacle size, obstacle position, etc. The road scene information includes: road type, road weather, etc.

[0047] Specifically, the model in this embodiment includes three task branches: a detection task branch, a semantic segmentation task branch, and a classification task branch. The detection task branch outputs the information of obstacles in the image features, the semantic segmentation task branch outputs the semantic information of each pixel in the image features, and the classification task branch outputs the type information of the road in the image features. The information of the target obstacle is determined based on the information and semantic information of the obstacle, and the target road scene information is determined based on the type information and semantic information of the road.

[0048] This embodiment extracts the image frames to be identified from the video stream, inputs the image frames to be identified into a trained recognition model, and obtains the target object information and target scene information output by the recognition model. The recognition model is used to identify the image feature extraction network that extracts the image features of the image frames to be identified. The recognition model also includes three task branches: a detection task branch for outputting the detection information of the object in the image frame, a semantic segmentation task branch for outputting the semantic information of the pixel points in the image frame, and a classification task branch for outputting the scene type information in the image frame. By performing multi-task processing on the same image feature at the same time, data processing time is saved. Finally, the target object information is determined based on the detection information and semantic information, and the target scene information is determined based on the scene type information and semantic information. The results output by the three task branches refer to each other, making the output related information more accurate.

[0049] How to optimize the recognition model is a continuous research for those skilled in the art. The recognition model of this application has been trained before use. Figure 4 , Figure 4 is a flowchart of the steps of constructing a recognition model shown in another exemplary embodiment of the present application, based on Figure 3 The identification method shown in FIG. 1 includes at least steps S410 to S430, which are described in detail as follows:

[0050] S410: Construct an initial recognition model, which includes an image feature extraction network, a detection task branch, a semantic segmentation task branch, and a classification task branch.

[0051] S410 is exemplified as follows: construct the detection task branch, semantic segmentation task branch and classification task branch in the recognition model. Among them, first construct an image feature extraction network, that is, a feature backbone network, which includes M blocks (network blocks). The downsampling ratio of each block of the feature backbone network can be 1 or 2, and the final downsampling ratio of the feature backbone network is 8; the detection task branch includes N detection heads, except that the output resolution of the first detection head is the resolution of the feature backbone network, and the resolution of each of the remaining detection heads is 1 / 2 of the previous detection head; the semantic segmentation task branch includes an S-layer simple neural network; the classification task branch includes an L-layer simple neural network; M, N, S and L are all integers. At the same time, a polling mechanism training method is established. During single-step training, only the training data of a single task branch in the recognition model is fed, and the three task branches are trained alternately and the parameters required for each task branch are updated.

[0052] Specifically, if Figure 5 As shown, Figure 5This is a schematic diagram of the structure of a recognition model, illustrating another exemplary embodiment of the present application. The block structure of a single feature network backbone includes: a first convolutional layer, six feature layers, and a fusion layer. The first convolutional layer has a stride of 1 or 2, c channels, and a kernel size of 3*3. The feature layers consist of three normal convolutional layers and three dilated convolutional layers, with 1 / 2c channels, a dilation scale of d, and a kernel size of 3*3. The fusion layer has a kernel size of 1*1 and c channels, where c and d are positive integers.

[0053] Furthermore, the semantic segmentation task branch is an S-layer simple convolutional neural network, the last layer of which uses bilinear interpolation to upsample to the same resolution as the input. The classification task branch is an L-layer simple convolutional neural network, the last layer of which has k channels and is pooled into k neurons through a global pooling, where k is the total number of categories. The detection branch consists of multiple detection heads of different scales, and performs anchor box regression through post-processing similar to YOLO. Here, S, L, and K are integers.

[0054] S420: The image frame to be trained and identified is input into the initial recognition model, and the image feature extraction network identifies and extracts features of the image frame to be trained to obtain the image features to be trained; the detection task branch outputs the first detection information of the object in the image frame corresponding to the image features to be trained, the semantic segmentation task branch outputs the first semantic information of the pixel points in the image frame corresponding to the image features to be trained, and the classification task branch outputs the first scene type information in the image frame corresponding to the image features.

[0055] For example, after the image frame to be trained and identified is input into the initial recognition model, the image feature extraction network will extract its image features, and the corresponding task branch will output the relevant information in the image frame corresponding to the image feature to be trained. Subsequently, the parameters of the task branch can be modified according to their respective corresponding information.

[0056] S430: Modify the initial recognition model according to the first detection information, the first semantic information, and the first scene information to obtain a trained recognition model.

[0057] S430 is exemplarily explained, where the detection task branch is corrected according to the first detection information, the semantic segmentation task branch is corrected according to the first semantic information, and the classification task branch is corrected according to the first scene information, thereby obtaining a trained recognition model.

[0058] This embodiment further defines the steps of constructing an initial recognition model and training each task branch in the initial model to obtain a trained recognition model. The trained image frame is simultaneously fed into a feature extraction network, and the extracted image features are fed into the three task branches to obtain corresponding information. The task branches are then trained based on the corresponding information, making the training process more refined and ultimately improving the accuracy of the recognition model.

[0059] Furthermore, another exemplary embodiment of the present application provides a specific correction method for the recognition model, please refer to Figure 6 , Figure 6 This is a flowchart of another exemplary embodiment of the present application showing a process of modifying a recognition model based on a loss function. Based on the above S430, it also includes at least S610 to S620, which are described in detail as follows:

[0060] S610: Determine a detection information loss function value based on the first detection information and the first standard detection information; determine a semantic information loss function value based on the first semantic information and the first semantic standard information; determine a scene type information loss function value based on the first scene type information and the first standard scene type information.

[0061] Each task branch has a corresponding standard information value. The first detection information, the first semantic information, and the first scene type information are respectively compared with their corresponding standard information values ​​to obtain their respective loss function values.

[0062] S620: Modify the initial recognition model based on the detection information loss function value, the semantic information loss function value, and the scene type information loss function value to obtain a trained recognition model.

[0063] The parameters of the corresponding task branch are corrected according to the corresponding loss function value to achieve the purpose of correcting the initial recognition model, thereby obtaining a trained recognition model.

[0064] This embodiment limits the initial recognition model to be modified according to the loss function value of each task branch, thereby obtaining a trained recognition model, so that the trained recognition model has higher accuracy.

[0065] In order to make the correction of the recognition model more theoretical and scientific, another exemplary embodiment of the present application introduces a back propagation value to assist in correcting the recognition model. For details, please refer to Figure 7 , Figure 7 This is a flowchart of another exemplary embodiment of the present application showing a process of correcting a recognition model based on back-propagation values. Based on the above S620, it also includes at least S710 to S740, which are described in detail as follows:

[0066] S710: Calculate a first back propagation value based on the detection information loss function value and a first dynamic modulation factor corresponding to the detection task branch.

[0067] S720: Calculate a second back propagation value based on the semantic information loss function value and the second dynamic modulation factor corresponding to the semantic segmentation task branch.

[0068] S730: Calculate a third back propagation value based on the scene type information loss function value and the third dynamic modulation factor corresponding to the classification task branch.

[0069] This embodiment calculates the value of directional propagation based on the dynamic modulation factor and loss function value corresponding to each task branch, updates all neural network parameters in the recognition model involved through the backpropagation mechanism, and uses the trained neural network parameters during inference to simultaneously perform detection of three tasks.

[0070] In particular, the relationship between the dynamic modulation factor and the loss function value is Lbi = 1 / Lsi*Lci, where Lbi is the loss function value of the i-th task backpropagated in the multi-task network, Lsi is the loss function value after the single task network is fully converged, and Lci is the loss function value calculated for the current number of steps of the task.

[0071] S740: Update the configuration parameters in the initial recognition model according to the first back-propagation value, the second back-propagation value, and the third back-propagation value to obtain a trained recognition model.

[0072] For example, the back propagation values ​​corresponding to each task branch are used to update the configuration parameters in the initial model to obtain a trained recognition model. In particular, Figure 8 As shown, Figure 8 This is a schematic diagram of edge-end quantization perception training for a recognition model shown in another exemplary embodiment of the present application. The present application creatively gives the board-end INT8 result to the PC-end FP32 model to calculate the loss function, thereby quantizing the trained neural network parameters through post-training and training perception quantization to obtain an INT8 tensor operation model that can be deployed on an edge computing processor; using the INT8 tensor operation model, the edge computing processes the image frame information in the video stream while obtaining detection information, semantic information and scene type information, determines the target object information based on the detection information and semantic information, and determines the target scene information based on the classification information and semantic information. Specifically, the image feature samples in the image feature sample set are input into the INT8 tensor operation model to obtain a prediction result, and the prediction result is compared and calculated with the preset standard value in the image feature sample set to obtain the loss function value, and then the corresponding back propagation value is obtained according to the loss function value to update the parameters to obtain the FP32 model as the trained recognition model of this embodiment.

[0073] This embodiment further introduces back-propagation values ​​to correct the parameters of the initial recognition model, further determines the corresponding back-propagation values ​​based on the loss function values, and finally corrects the parameters of the initial recognition model based on the back-propagation values ​​to obtain a trained recognition model, making the recognition model correction process more theoretical and scientific, and making the recognition model more accurate.

[0074] In another exemplary embodiment, the configuration parameters in the initial recognition model are updated according to the first back propagation value, the second back propagation value, and the third back propagation value to obtain a trained recognition model, including: training the parameters in the detection task branch according to the first back propagation value to obtain a trained detection task branch; training the parameters in the semantic segmentation task branch according to the second back propagation value to obtain a trained semantic segmentation task branch; training the parameters in the classification task branch according to the third back propagation value to obtain a trained classification task branch; and obtaining a trained recognition model.

[0075] This embodiment limits the training of parameters in the corresponding task branches according to the back-propagation values. The simultaneous training of multiple task parameters saves the time of model optimization, and updates the trained classification task branches in the recognition model to obtain a trained recognition model. The training is more refined, and the recognition accuracy of the trained recognition model is higher.

[0076] In another exemplary embodiment, how to determine the category of the target object is further defined. The detection information includes the position of the detection regression box in the image frame and the category of the predicted object. The semantic information includes the number and position of pixels corresponding to the predicted object in the image frame, and the semantic type corresponding to the pixels in the image frame. Based on S320, this step specifically includes S810 to S820, which are described in detail below:

[0077] S810: Determine the number of pixels corresponding to the predicted object within the detection regression frame according to the position of the detection regression frame.

[0078] In this embodiment, the detection regression frame is a target frame within the image features included in the detection information. It is used to select the predicted object, that is, to determine the position of the predicted object within the image features. Knowing the position of the detection regression frame within the image features, the number of pixels corresponding to the predicted object within the detection regression frame is counted to obtain the number of pixels corresponding to the predicted object within the regression frame.

[0079] For example, the detection information output by the detection task branch includes the detection regression box B in the image features i (X1, Y1, X2, Y2) and the category C of the predicted object, then the detection regression box B iThe coordinates of can determine its position in the image feature and the area of ​​the framed region, and the pixel points of the C-type prediction object in the framed region are counted to obtain the number of pixel points of the C-type prediction object.

[0080] S820: If the number of pixels corresponding to the predicted object is greater than the number of pixels of the preset category object corresponding to the category of the predicted object, the position of the detection regression box is determined to be the position of the target object, and the category of the predicted object is the category of the target object.

[0081] Exemplarily, the number of pixel points of the preset category object corresponding to the C-type prediction object is 10. If the number of pixel points corresponding to the C-type prediction object is 15, that is, the number of pixel points corresponding to the prediction object is greater than the number of pixel points of the preset category object corresponding to the category of the prediction object, then the C-type prediction object can be determined as the target object, and the position of the detection regression box is the position of the target object.

[0082] This embodiment limits the information contained in the detection information, and further clarifies how to use the position of the detection regression box and the category of the predicted object in the image features included in the detection information to determine the category of the target object, and at the same time determine the exact position of the target object in the image frame to be identified, making the target object identification process more accurate.

[0083] In another exemplary embodiment, how to determine the scene type of the target scene is further defined. Based on S320, the semantic type includes: scene type and object type; the scene type information includes the scene type in the image frame and the scene probability value corresponding to the scene type. S320 includes S910 to S930, which are described in detail below:

[0084] S910: If the scene type in the semantic type is the same as the scene type in the scene type information, determine that the scene type in the scene type information is the scene type of the target scene.

[0085] For example, if the scene type in the semantic type is D and the scene type in the scene type information is also D, then the scene type of the target scene finally output by the recognition model is D.

[0086] S920: If the scene type in the semantic type is different from the scene type in the scene type information, and the scene probability value corresponding to the scene type is greater than the preset scene probability threshold, determine that the scene type in the scene type information is the scene type of the target scene.

[0087] The preset scene probability threshold in this embodiment is a threshold pre-set in the recognition model, which is used to determine the scene type of the target scene. This embodiment does not specifically limit the setting process of the preset scene probability threshold and its threshold value.

[0088] Exemplarily, the scene type in the scene type information is B, the corresponding B scene probability is 0.5, the scene type in the semantic type is A, and the preset scene probability threshold is 0.4, then the scene type of the target scene is determined to be B.

[0089] S930: If the scene type in the semantic type is different from the scene type in the scene type information, and the scene probability value corresponding to the scene type is less than or equal to the preset scene probability threshold, then determine that the scene type in the semantic type corresponding to the pixel point in the image feature is the scene type of the target scene.

[0090] Exemplarily, the scene type in the scene type information is B, the corresponding B scene probability is 0.4, the scene type in the semantic type is A, and the preset scene probability threshold is 0.4, then the scene type of the target scene is determined to be A.

[0091] This embodiment compares the scene type in the semantic type with the scene type in the scene type information for similarity and introduces a preset scene probability threshold, thereby accurately determining the scene type of the target scene in the image frame to be identified.

[0092] In another exemplary embodiment, a method for acquiring a preceding image frame to be recognized is defined. Based on the above S310, S1010 to S1020 are also included, which are described in detail below:

[0093] S1010: Acquire a video stream, where the video stream includes multiple image frames.

[0094] S1020: Detect whether a target object exists in each image frame respectively, and determine the image frame in which the target object is detected as the image frame to be identified.

[0095] For example, if an image frame is a blank image frame, that is, there is no object in the blank image frame, there is definitely no target object. If the image frame is used to identify the target object and scene type, the data processing capacity of the recognition model will obviously increase.

[0096] This embodiment pre-checks each image frame in the video stream, avoiding useless image frames from being input into the recognition model as image frames to be recognized, causing the recognition model to perform useless / meaningless recognition, avoiding the waste of data processing resources of the recognition model, and improving the accuracy of the recognition results.

[0097] Another aspect of the present application also provides an identification device, such as Figure 9 As shown, Figure 9 FIG. 1 is a schematic diagram illustrating the structure of an identification device according to an exemplary embodiment of the present application. The identification device includes:

[0098] The extraction module 910 is configured to extract image frames to be identified from the video stream.

[0099] The output module 930 is configured to input the image frame to be identified into a trained recognition model to obtain the target object information and target scene information output by the recognition model; wherein the recognition model includes an image feature extraction network for identifying and extracting image features of the image frame to be identified, a detection task branch for outputting detection information of the object in the image frame, a semantic segmentation task branch for outputting semantic information of pixel points in the image frame, and a classification task branch for outputting scene type information in the image frame, determining the target object information based on the detection information and semantic information, and determining the target scene information based on the scene type information and semantic information.

[0100] In another embodiment, the identification device further comprises:

[0101] The building module is configured to build an initial recognition model, which includes an image feature extraction network, a detection task branch, a semantic segmentation task branch, and a classification task branch.

[0102] The training module is configured to input the image frame to be trained and identified into the initial recognition model, and the image feature extraction network identifies and extracts features of the image frame to be trained to obtain the image features to be trained; the detection task branch outputs the first detection information of the object in the image frame corresponding to the image feature to be trained, the semantic segmentation task branch outputs the first semantic information of the pixel point in the image frame corresponding to the image feature to be trained, and the classification task branch outputs the first scene type information in the image frame corresponding to the image feature.

[0103] The correction module is configured to correct the initial recognition model according to the first detection information, the first semantic information, and the first scene information to obtain a trained recognition model.

[0104] In another embodiment, the correction module includes:

[0105] The loss function value determination unit is configured to determine the detection information loss function value based on the first detection information and the first standard detection information; determine the semantic information loss function value based on the first semantic information and the first semantic standard information; and determine the scene type information loss function value based on the first scene type information and the first standard scene type information.

[0106] The loss function correction unit is configured to correct the initial recognition model based on the detection information loss function value, the semantic information loss function value, and the scene type information loss function value to obtain a trained recognition model.

[0107] In another embodiment, the loss function modification unit includes:

[0108] The first section is configured to calculate a first back propagation value based on the detection information loss function value and the first dynamic modulation factor corresponding to the detection task branch.

[0109] The second module is configured to calculate the second backpropagation value based on the semantic information loss function value and the second dynamic modulation factor corresponding to the semantic segmentation task branch.

[0110] The third section is configured to calculate a third back propagation value based on the scene type information loss function value and the third dynamic modulation factor corresponding to the classification task branch.

[0111] The update section is configured to update the configuration parameters in the initial recognition model according to the first back-propagation value, the second back-propagation value, and the third back-propagation value to obtain a trained recognition model.

[0112] In another embodiment, the output module 930 includes:

[0113] The pixel determination unit is configured to determine the number of pixel points corresponding to the predicted object in the detection regression box according to the position of the detection regression box.

[0114] The target object category determination unit is configured to determine the position of the detection regression box as the position of the target object and the category of the predicted object as the category of the target object if the number of pixel points corresponding to the predicted object is greater than the number of pixel points of the preset category object corresponding to the category of the predicted object.

[0115] In another embodiment, the semantic type includes: scene type and object type; the output module 930 includes:

[0116] The first type unit is configured to determine that the scene type in the scene type information is the scene type of the target scene if the scene type in the semantic type is the same as the scene type in the scene type information.

[0117] The second type unit is configured to determine that the scene type in the scene type information is the scene type of the target scene if the scene type in the semantic type is different from the scene type in the scene type information and the scene probability value corresponding to the scene type is greater than a preset scene probability threshold.

[0118] The third type unit is configured to determine that the scene type in the semantic type corresponding to the pixel point in the image feature is the scene type of the target scene if the scene type in the semantic type is different from the scene type in the scene type information and the scene probability value corresponding to the scene type is less than or equal to a preset scene probability threshold.

[0119] In another embodiment, the extraction module 910 includes:

[0120] The acquisition unit is configured to acquire a video stream, where the video stream includes a plurality of image frames.

[0121] The detection unit is configured to respectively detect whether a target object exists in each image frame, and determine the image frame in which the target object is detected as the image frame to be identified.

[0122] In another embodiment, the identification device further comprises:

[0123] The correction module is configured to correct the initial recognition model to obtain a corrected recognition model.

[0124] The quantization module is configured to perform quantization perception training of the INT8 edge computing module on the modified recognition model to obtain a trained recognition model.

[0125] It should be noted that the identification device provided in the above embodiment and the identification method provided in the above embodiment belong to the same concept, and the specific manner in which each module and unit performs operations has been described in detail in the method embodiment and will not be repeated here.

[0126] Another aspect of the present application provides an electronic device, including: a controller; and a memory for storing one or more programs, which, when executed by the controller, execute the methods identified in the above embodiments.

[0127] See also Figure 10 , Figure 10 1 is a schematic diagram of the structure of a computer system of an electronic device shown in an exemplary embodiment of the present application, which shows a schematic diagram of the structure of a computer system of an electronic device suitable for implementing an embodiment of the present application.

[0128] It should be noted that Figure 10 The computer system 1000 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0129] like Figure 10As shown, the computer system 1000 includes a central processing unit (CPU) 1001, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 1002 or the program loaded from the storage part 1008 into the random access memory (RAM) 1003, such as executing the method in the above embodiment. Various programs and data required for system operation are also stored in the RAM 1003. The CPU 1001, ROM 1002 and RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0130] The following components are connected to the I / O interface 1005: an input section 1006 including a keyboard, a mouse, and the like; an output section 1007 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 1008 including a hard disk and the like; and a communication section 1009 including a network interface card such as a LAN (Local Area Network) card or a modem. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. Removable media 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 1010 as needed, so that computer programs read therefrom can be installed into the storage section 1008 as needed.

[0131] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 1009, and / or installed from a removable medium 1011. When the computer program is executed by the central processing unit (CPU) 1001, the various functions defined in the system of the present application are executed.

[0132] It should be noted that the computer-readable medium shown in the embodiments of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, which carries a computer-readable computer program. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. A computer program embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0133] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. Among them, each box in the flowchart or block diagram can represent a module, program segment, or part of the code, and the above-mentioned module, program segment, or part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0134] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. In some cases, the names of these units do not constitute limitations on the units themselves.

[0135] Another aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the aforementioned identification method. The computer-readable storage medium may be included in the electronic device described in the above embodiments, or may exist independently and not be incorporated into the electronic device.

[0136] Another aspect of the present application further provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the identification method provided in each of the above embodiments.

[0137] According to one aspect of an embodiment of the present application, a computer system is further provided, including a central processing unit (CPU), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) or a program loaded from a storage portion into a random access memory (RAM), such as executing the method in the above embodiment. Various programs and data required for system operation are also stored in the RAM. The CPU, ROM, and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.

[0138] The following components are connected to the I / O interface: an input section including a keyboard, mouse, etc.; an output section including a cathode ray tube (CRT), liquid crystal display (LCD), and speakers; a storage section including a hard disk; and a communication section including a network interface card such as a LAN (Local Area Network) card and a modem. The communication section performs communication processing via a network such as the Internet. A drive is also connected to the I / O interface as needed. Removable media such as magnetic disks, optical disks, magneto-optical disks, semiconductor memories, etc. are installed in the drive as needed so that computer programs read from them can be installed into the storage section as needed.

[0139] The above content is only a preferred exemplary embodiment of the present application and is not intended to limit the implementation scheme of the present application. Ordinary technicians in this field can easily make corresponding changes or modifications based on the main ideas and spirit of the present application. Therefore, the scope of protection of the present application shall be based on the scope of protection required by the claims.

Claims

1. A recognition method, characterized in that: The identification method comprises: Extracting image frames to be identified from the video stream; Inputting an image frame to be identified into a trained recognition model to obtain target object information and target scene information output by the recognition model; wherein the recognition model includes an image feature extraction network for identifying and extracting image features of the image frame to be identified, a detection task branch for outputting detection information of the object in the image frame, a semantic segmentation task branch for outputting semantic information of pixels in the image frame, and a classification task branch for outputting scene type information in the image frame, determining the target object information based on the detection information and the semantic information, and determining the target scene information based on the scene type information and the semantic information; The detection information includes the position of the detection regression box in the image frame and the category of the predicted object, the semantic information includes the number and position of pixels corresponding to the predicted object in the image frame, and the semantic type corresponding to the pixels in the image frame, the target object information includes the category and position of the target object, and the information of the target object determined based on the detection information and the semantic information includes: Determine the number of pixels corresponding to the predicted object within the detection regression frame according to the position of the detection regression frame; If the number of pixels corresponding to the predicted object is greater than the number of pixels of the preset category object corresponding to the category of the predicted object, the position of the detection regression box is determined to be the position of the target object, and the category of the predicted object is the category of the target object.

2. The identification method according to claim 1, characterized in that The identification method further includes: Constructing an initial recognition model, wherein the initial recognition model includes the image feature extraction network, the detection task branch, the semantic segmentation task branch, and the classification task branch; Inputting the image frame to be trained and identified into the initial recognition model, the image feature extraction network recognizes and extracts features from the image frame to be trained and obtains image features to be trained; the detection task branch outputs first detection information of an object in the image frame corresponding to the image feature to be trained, the semantic segmentation task branch outputs first semantic information of a pixel point in the image frame corresponding to the image feature to be trained, and the classification task branch outputs first scene type information in the image frame corresponding to the image feature; The initial recognition model is modified according to the first detection information, the first semantic information, and the first scene type information to obtain the trained recognition model.

3. The identification method according to claim 2, characterized in that The modifying the initial recognition model according to the first detection information, the first semantic information, and the first scene type information to obtain the trained recognition model includes: Determining a detection information loss function value based on the first detection information and the first standard detection information; determining a semantic information loss function value based on the first semantic information and the first semantic standard information; determining a scene type information loss function value based on the first scene type information and the first standard scene type information; The initial recognition model is modified based on the detection information loss function value, the semantic information loss function value, and the scene type information loss function value to obtain the trained recognition model.

4. The identification method according to claim 3, characterized in that The correcting the initial recognition model based on the detection information loss function value, the semantic information loss function value, and the scene type information loss function value to obtain the trained recognition model includes: A first back propagation value is calculated based on the detection information loss function value and the first dynamic modulation factor corresponding to the detection task branch; A second back propagation value is calculated based on the semantic information loss function value and the second dynamic modulation factor corresponding to the semantic segmentation task branch; A third back propagation value is calculated based on the scene type information loss function value and the third dynamic modulation factor corresponding to the classification task branch; The configuration parameters in the initial recognition model are updated according to the first back-propagation value, the second back-propagation value, and the third back-propagation value to obtain the trained recognition model.

5. The identification method according to claim 4, characterized in that: The semantic type includes: a scene type and an object type; the scene type information includes the scene type in the image frame and a scene probability value corresponding to the scene type; and determining the target scene information based on the scene type information and the semantic information includes: If the scene type in the semantic type is the same as the scene type in the scene type information, determining that the scene type in the scene type information is the scene type of the target scene; If the scene type in the semantic type is different from the scene type in the scene type information, and the scene probability value corresponding to the scene type is greater than a preset scene probability threshold, determining that the scene type in the scene type information is the scene type of the target scene; If the scene type in the semantic type is different from the scene type in the scene type information, and the scene probability value corresponding to the scene type is less than or equal to the preset scene probability threshold, then the scene type in the semantic type corresponding to the pixel point in the image feature is determined to be the scene type of the target scene.

6. The identification method according to any one of claims 1 to 5, characterized in that: The step of extracting the image frame to be identified from the video stream includes: Acquire a video stream, where the video stream includes a plurality of image frames; Each image frame is detected to determine whether the target object exists, and the image frame in which the target object is detected is determined as the image frame to be identified.

7. The identification method according to claim 3, characterized in that: Before obtaining the trained recognition model, the method further includes: Modifying the initial recognition model to obtain a modified recognition model; The quantized perception training of the INT8 edge computing module is performed on the modified recognition model to obtain a trained recognition model.

8. An identification device, characterized in that: include: An extraction module is configured to extract image frames to be identified from a video stream; an output module configured to input image features of an image frame to be recognized into a trained recognition model, and obtain target object information and target scene information output by the recognition model; wherein the recognition model includes a detection task branch for outputting detection information of an object in the image frame, a semantic segmentation task branch for outputting semantic information of pixels in the image frame, and a classification task branch for outputting scene type information in the image frame, determining the target object information based on the detection information and the semantic information, and determining the target scene information based on the scene type information and the semantic information; The detection information includes the position of the detection regression box in the image frame and the category of the predicted object, the semantic information includes the number and position of pixels corresponding to the predicted object in the image frame, and the semantic type corresponding to the pixels in the image frame, the target object information includes the category and position of the target object, and the information of the target object determined based on the detection information and the semantic information includes: Determine the number of pixels corresponding to the predicted object within the detection regression frame according to the position of the detection regression frame; If the number of pixels corresponding to the predicted object is greater than the number of pixels of the preset category object corresponding to the category of the predicted object, the position of the detection regression box is determined to be the position of the target object, and the category of the predicted object is the category of the target object.

9. A computer-readable storage medium, characterized in that Computer-readable instructions are stored thereon, and when the computer-readable instructions are executed by a processor of a computer, the computer is caused to execute the identification method according to any one of claims 1 to 7.

Images