Joint perception model training, joint perception method, device, equipment and medium

Abstract

The present disclosure provides a joint perception model training, joint perception method, device, equipment and medium, which relate to the field of artificial intelligence technology, especially computer vision, image recognition and deep learning technology. The specific implementation scheme is: obtain a sample image and a perception label of the sample image; obtain a preset joint perception model; the joint perception model includes a feature extraction network and a joint perception network; extract features of the sample image through the feature extraction network to obtain target sample features; perform joint perception according to the target sample features through the joint perception network to obtain perception prediction results; train the preset joint perception model according to the perception prediction results and the perception labels; wherein the joint perception includes performing at least two perception tasks. According to the technology disclosed in the present disclosure, the amount of data calculation in the joint perception process is reduced, and the computing efficiency is improved.

Description

[0001] The present disclosure is a divisional application of the invention patent application with application number 202210218587.8. Technical Field

[0002] The present disclosure relates to the field of artificial intelligence technology, in particular to computer vision, image recognition and deep learning technology, and specifically to a joint perception model training, a joint perception method, a device, a equipment and a medium. Background Art

[0003] With the continuous development of computer science, artificial intelligence technology has emerged. Artificial intelligence is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence.

[0004] Image processing based on artificial intelligence technology is of great significance for the construction of smart transportation, smart security and smart cities. Summary of the invention

[0005] The present disclosure provides a joint perception model training, a joint perception method, an apparatus, a device and a medium.

[0006] According to one aspect of the present disclosure, a joint perception model training method is provided, comprising:

[0007] Obtain sample images and perceptual labels of the sample images;

[0008] Acquire a preset joint perception model; the joint perception model includes a feature extraction network and a joint perception network;

[0009] Extract features from sample images through a feature extraction network to obtain target sample features;

[0010] The joint perception network performs joint perception according to the characteristics of the target samples to obtain the perception prediction results;

[0011] The preset joint perception model is trained according to the perception prediction results and the perception labels; wherein the joint perception includes performing at least two perception tasks.

[0012] According to another aspect of the present disclosure, a joint sensing method is also provided, including:

[0013] Get the predicted image;

[0014] Input the predicted image into the target joint perception model to obtain the perception prediction result;

[0015] Among them, the target joint perception model is obtained based on an initial joint perception model trained by any one of the joint perception model training methods provided in the embodiments of the present disclosure.

[0016] According to another aspect of the present disclosure, there is also provided an electronic device, including:

[0017] at least one processor; and

[0018] a memory communicatively connected to at least one processor; wherein,

[0019] The memory stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor so that the at least one processor can execute any one of the joint perception model training methods provided by the embodiments of the present disclosure, and / or any one of the joint perception methods provided by the embodiments of the present disclosure.

[0020] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is also provided, wherein the computer instructions are used to enable a computer to execute any one of the joint perception model training methods provided by the embodiments of the present disclosure, and / or any one of the joint perception methods provided by the embodiments of the present disclosure.

[0021] According to the technology disclosed in the present invention, the amount of data calculation in the joint perception process is reduced and the calculation efficiency is improved.

[0022] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0024] Figure 1 is a flow chart of a joint perception model training method provided by an embodiment of the present disclosure;

[0025] Figure 2 is a structural diagram of a joint perception model provided by an embodiment of the present disclosure;

[0026] Figure 3A is a structural diagram of a feature extraction network provided by an embodiment of the present disclosure;

[0027] Figure 3B is a structural schematic diagram of a first multi-channel module provided by an embodiment of the present disclosure;

[0028] Figure 3C is a schematic diagram of the structure of a first feature extraction subnetwork provided by an embodiment of the present disclosure;

[0029] Figure 3D is a schematic diagram of the structure of a first feature extraction subnetwork provided by an embodiment of the present disclosure;

[0030] Figure 4A is a structural diagram of a joint sensing network provided by an embodiment of the present disclosure;

[0031] Figure 4B is a schematic structural diagram of a detection head module provided by an embodiment of the present disclosure;

[0032] Figure 4C is a structural schematic diagram of a second multi-channel module provided by an embodiment of the present disclosure;

[0033] Figure 4D is a structural diagram of a second feature extraction subnetwork provided by an embodiment of the present disclosure;

[0034] Figure 4E is a structural diagram of a second feature extraction subnetwork provided by an embodiment of the present disclosure;

[0035] Figure 5 is a flow chart of a joint perception method provided by an embodiment of the present disclosure;

[0036] Fig. 6A is a structural diagram of the first feature extraction subnetwork after re-parameterization provided by an embodiment of the present disclosure;

[0037] Figure 6B is a structural diagram of a re-parameterized second feature extraction sub-network provided in an embodiment of the present disclosure;

[0038] Figure 7 is a structural diagram of a joint perception model training device provided by an embodiment of the present disclosure;

[0039] Figure 8 is a structural diagram of a joint sensing device provided by an embodiment of the present disclosure;

[0040] Fig. 9 It is a block diagram of an electronic device used to implement the joint perception model training method and / or the joint perception method of the embodiments of the present disclosure. DETAILED DESCRIPTION

[0041] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0042] The disclosed embodiments are applicable to the case of performing multi-task joint perception of an image under at least one category of target, and provide technical support for the construction of smart cities, smart transportation, and smart security. The so-called multi-task joint perception may include the execution of at least two of the target detection task, the target attribute recognition task, and the target search feature extraction task. Among them, the target is the object to be labeled in the image, and the target detection, attribute recognition, or search feature extraction is performed on the object to be labeled.

[0043] It should be noted that since the joint perception process of the image needs to be implemented based on the joint perception model, in order to facilitate understanding, the training process of the joint perception model is first described in detail.

[0044] Figure 1 This is a flowchart of a joint perception model training method provided by an embodiment of the present disclosure, which is applicable to application scenarios for training a joint perception model with a multi-task joint perception function. The joint perception model training method provided by an embodiment of the present disclosure can be performed by a joint perception model training device, which can be implemented in software and / or hardware and specifically configured in an electronic device.

[0045] like Figure 1 A joint perception model training method shown includes:

[0046] S101: Obtain a sample image and a perceptual label of the sample image.

[0047] The sample image is the image used in the model training process. The perceptual label of the sample image is used to characterize the theoretical result after joint perception of the sample image, which may include standard perceptual results corresponding to different perceptual tasks. The joint perception may include at least two perceptual tasks of target detection, target attribute recognition, and target search feature extraction.

[0048] Exemplarily, the standard perception results corresponding to the target detection task may include the position of the annotation box of the object to be annotated, for example, may include at least one of the coordinates, size, scale and confidence of the annotation box; the standard perception results corresponding to the attribute recognition task may include at least one attribute category of the object to be annotated, for example, a vehicle may include vehicle color, vehicle type, etc., a face may include gender, etc., and a human body may include clothing color, etc.; the standard perception results corresponding to the search feature extraction task may include feature vectors used for image retrieval, etc.

[0049] In an optional embodiment, the sample image may be a picture frame obtained by performing frame extraction processing on the video stream data.

[0050] In order to reduce the impact of sample differences on the model training process, in an optional embodiment, different sample images may be preprocessed uniformly.

[0051] Exemplarily, the sample images may be scaled to unify the image sizes of different sample images. Exemplarily, the sample images may be normalized, for example, the mean of the three primary colors RGB (red, green, and blue) is uniformly subtracted from each sample image, so as to enhance the robustness of the trained model.

[0052] S102, obtaining a preset joint perception model; the joint perception model includes a feature extraction network and a joint perception network.

[0053] S103: extract features from the sample image through a feature extraction network to obtain target sample features.

[0054] S104. Perform joint perception according to the target sample characteristics through a joint perception network to obtain a perception prediction result; wherein the joint perception includes performing at least two perception tasks.

[0055] S105: training a preset joint perception model according to the perception prediction results and the perception labels.

[0056] Among them, the joint perception model can be constructed using a machine learning model or a deep learning model.

[0057] The perception prediction result may be a prediction result corresponding to different perception tasks. For example, the perception prediction result may include at least one of the predicted position of the annotation box of the object to be annotated in the target detection task, at least one attribute category of the object to be annotated in the attribute recognition task, and the predicted feature vector for image search in the search feature extraction task.

[0058] See also Figure 2 The structural diagram of the joint perception model shown in the figure. The joint perception model includes a feature extraction network and a joint perception network; the feature extraction network extracts features from the sample image to obtain target sample features; the joint perception network performs joint perception based on the target sample features to obtain perception prediction results; according to the difference between the perception prediction results and the perception labels, the network parameters in the joint perception model are optimized until the model performance of the joint perception model tends to be stable or the number of training samples reaches a preset threshold, and a trained joint perception model is obtained for subsequent joint perception prediction of the input image.

[0059] It should be noted that by sharing the target sample features extracted by the feature extraction network for different perception tasks in the joint perception network, the feature extraction network in the joint perception model learns the ability to extract visual features corresponding to different perception tasks during the model training process. There is no need to set up independent feature extraction networks for different perception tasks, which saves a lot of computing resources and improves computing efficiency. At the same time, the features between different perception tasks enhance each other, improving the feature performance of a single perception task.

[0060] Among them, the target sample features include visual features corresponding to different perception tasks, which are used as data support for subsequent joint perception by the joint perception network.

[0061] In an optional embodiment, the feature extraction network can be implemented based on a feature pyramid network, thereby performing multi-scale feature extraction and improving the feature richness of the target sample features.

[0062] Although the feature extraction network has learned the ability to extract visual features corresponding to different perception tasks, there may be a loss of feature information corresponding to some perception tasks, resulting in a reduction in the richness and comprehensiveness of the extracted features, which in turn affects the accuracy of the perception results of the joint perception model.

[0063] In order to further improve the joint perception capability of the model, in another optional embodiment, the feature extraction network can be improved to further enhance the joint extraction capability of multi-scale features under different perception tasks in the feature extraction network.

[0064] See also Figure 3A The schematic diagram of the feature extraction network shown can further refine the feature extraction network to include a base module and at least two first multi-channel modules. The base module downsamples the sample image to obtain initial sample features of different scales; the first multi-channel module extracts features of different perception tasks from the initial sample features to obtain target sample features at the corresponding scale of the initial sample features.

[0065] That is, the sample image is downsampled through the base model to obtain initial sample features of different scales; for the initial sample features of any scale, the features of different perception tasks are extracted from the initial sample features of the scale through the first multi-channel module corresponding to the scale to obtain the target sample features at the scale.

[0066] in, Figure 3AAn example is given in which the base module is used to downsample the sample image step by step to obtain five initial sample features of different scales C1-C5; C3-C5 are processed by the corresponding first multi-channel modules respectively to obtain target sample features P3-P5. It should be noted that the above only gives the number of initial sample features and the number of target sample features by way of example, which should not be understood as a specific limitation on the two. Those skilled in the art can adjust the corresponding number according to actual needs.

[0067] It can be understood that by introducing the first multi-path module into the feature extraction network, the initial sample features at a single scale are respectively subjected to feature extraction for different perception tasks, so that the feature extraction network has the ability of multi-scale feature extraction under different perception tasks, thereby improving the richness and comprehensiveness of the extracted target sample features, and laying the foundation for improving the model accuracy of the joint perception model.

[0068] In an optional embodiment, the initial sample features of a single scale may be fused with target sample features of other scales to update the initial sample features of the single scale, thereby improving the richness of the target sample features of the single scale finally obtained after being processed by the corresponding first multi-channel module.

[0069] In a specific implementation, for the first multi-channel module processing process of adjacent scales, the target sample features of the smaller scale in the adjacent scales are fused with the initial sample features of the larger scale in the adjacent scales to update the initial sample features of the larger scale in the adjacent scale, thereby increasing the richness of the initial sample features of the larger scale, and further improving the accuracy of the output results of the target sample features of the larger scale.

[0070] It should be noted that joint perception can be performed based on target sample features of a single scale, and the perception prediction results of different scales can be fused to obtain the final perception prediction result; or optionally, target sample features of different scales can be fused, and the fusion results can be jointly perceived to obtain the final perception prediction result.

[0071] The specific structure of the first multi-path module in the feature extraction network will be described in detail below.

[0072] See also Figure 3BThe structural schematic diagram of the first multi-channel module shown in the figure, for the initial sample features of a single scale, the first multi-channel module can be further refined to include a first splitting subnetwork, a first feature extraction subnetwork and a first fusion subnetwork; the first splitting subnetwork splits the initial sample features according to the channel dimension to obtain the first sample features to be fused and the first sample features to be processed; the first feature extraction subnetwork extracts features from the first sample features to be processed to obtain the first target intermediate sample features; the first fusion subnetwork fuses the first sample features to be fused and the first target intermediate sample features to obtain the target sample features at this scale.

[0073] Exemplarily, by splitting the initial sample features according to the channel dimension, at least two groups of split results can be obtained. When splitting, the splitting can be performed randomly or sequentially according to the channel dimension. The present disclosure does not impose any restrictions on the specific splitting mechanism, and only needs to ensure that the splitting is performed according to the channel dimension. It should be noted that the present disclosure does not impose any restrictions on the number of channels of features in different splitting results.

[0074] In order to facilitate subsequent processing, the number of split results can be limited to two, that is, the initial sample feature is split to obtain the first sample feature to be fused and the first sample feature to be processed. In order to facilitate the splitting operation, the initial sample feature can be split into the first sample feature to be fused and the first sample feature to be processed by sequential splitting. Optionally, the number of channels of the first sample feature to be fused and the first sample feature to be processed can be the same.

[0075] It is understandable that the method of extracting features from the first sample feature to be processed in the initial sample feature instead of extracting features from the entire amount of initial sample features reduces the number of channels in the feature extraction process (when the number of channels of the first sample feature to be fused and the first sample feature to be processed is the same, the number of channels is halved), thereby reducing the amount of computation and memory usage in the feature extraction process. At the same time, by performing feature fusion on the first target intermediate sample feature obtained by extracting the first sample feature to be fused in the initial sample feature and the first sample feature to be processed, the number of gradient paths is increased (when the number of channels of the first sample feature to be fused and the first sample feature to be processed is the same, the number of gradient paths is doubled), thereby improving the learning ability of the trained model.

[0076] Taking the initial sample feature of shape (n,c,h,w) as an example, the processing process of the first multi-channel module is described. Where n is the number of sample images, c is the number of channels, h is the height of the sample image, and w is the width of the sample image. The initial sample feature is evenly split along the channel dimension to obtain the first sample feature to be fused and the first sample feature to be processed, both of which have the shape of (n,c / 2,h,w). The first feature extraction subnetwork extracts the first sample feature to be processed of shape (n,c / 2,h,w) to obtain the first target intermediate sample feature of shape (n,c / 2,h,w). Since the number of channels in the feature extraction process is halved, the computational complexity and video memory usage of the feature extraction process are reduced. By fusion of the first sample feature to be fused of shape (n,c / 2,h,w) and the first target intermediate sample feature of shape (n,c / 2,h,w), the target sample feature of shape (n,c,h,w) is obtained, and the number of gradient paths is doubled, thereby improving the learning ability of the trained model.

[0077] In order to further improve the comprehensiveness and richness of the features extracted by the first feature extraction subnetwork, in an optional embodiment, features of different dimensions may be extracted in the first feature extraction subnetwork.

[0078] See also Figure 3C The schematic diagram of the first feature extraction subnetwork shown in the figure can be set to include a first global perception module, a first local perception module and a first perception fusion module; the first global perception module performs global feature extraction on the first sample feature to be processed to obtain a first global intermediate sample feature; the first local perception module performs local feature extraction on the first sample feature to be processed to obtain a first local intermediate sample feature; the first perception fusion module performs feature fusion on the first global intermediate sample feature and the first local intermediate sample feature to obtain a first target intermediate sample feature.

[0079] It can be understood that by extracting features of the first sample to be processed from the global dimension and the local dimension respectively, and fusing the obtained first global intermediate sample features and the first local intermediate sample features, the first target intermediate sample features obtained after the fusion carry both overall context information and local details, thereby improving the richness and comprehensiveness of the first target intermediate sample features.

[0080] In an alternative embodiment, see Figure 3DThe schematic diagram of the first feature extraction subnetwork shown in the figure, the first global perception module in the first feature extraction subnetwork may include a pooling layer and an activation layer, wherein the pooling layer is used to perform global feature extraction on the first sample feature to be processed; the activation layer is used to perform nonlinear calculation on the global feature extraction result, thereby improving the nonlinear feature extraction capability of the first global perception module.

[0081] It should be noted that the embodiments of the present disclosure do not limit the specific pooling method used by the pooling layer and the specific activation function used by the activation layer, and can be set or adjusted according to actual needs and model training conditions. For example, the pooling layer can use average pooling, and the activation layer can use ReLU activation function.

[0082] In an optional embodiment, at least one first local perception branch may be set in the first local perception module according to actual needs to perform local feature extraction.

[0083] For example, see Figure 3D At least two first local perception branches can be set in the first local perception module of the first feature extraction subnetwork, and local feature extraction is performed on the first sample feature to be processed under different receptive fields through different first local perception branches to obtain the first local intermediate sample feature under the corresponding receptive field. Correspondingly, the first perception fusion module can first fuse the first local intermediate sample features under different receptive fields (for example, pixel superposition) to obtain the first local intermediate sample feature fusion result; fuse the first local intermediate sample feature fusion result with the first global intermediate sample feature (for example, channel multiplication) to obtain the first target intermediate sample feature.

[0084] It can be understood that by setting at least two first local perception branches to extract local features under different receptive fields, the richness of the first local intermediate sample features is improved, which helps to improve the subsequent labeling capabilities of objects to be labeled of different scales, and lays the foundation for the labeling of multi-category targets. At the same time, by extracting local features under different receptive fields, the extracted features can support different perception tasks, which helps to improve the perception accuracy of the joint perception model.

[0085] In a specific implementation, a convolution layer and a batch processing layer can be set in the first local perception branch. The convolution layer is used to extract local features under the corresponding receptive field according to the convolution kernel; the batch processing layer is used to normalize the extracted features, so as to assimilate the distribution of the first local intermediate sample features extracted by different first local perception branches, accelerate the convergence of the model, and improve the stability of the training process.

[0086] It should be noted that the convolution kernel scales used in different first local perception branches are different, so that different first local perception branches can perform local feature extraction under different receptive fields. Among them, the number of convolution kernels can be set or adjusted by technicians according to needs or experience values, and the type of convolution kernel can be specifically set according to the perception task of joint perception. Exemplarily, for target search feature extraction tasks, since the search process usually requires multi-granularity features, both large-scale convolution kernels and small-scale convolution kernels need to be set. Exemplarily, for attribute recognition tasks and target detection tasks, horizontal or vertical convolution kernels need to be set for different target categories. Specifically, 1*3 convolution kernels and 3*5 convolution kernels can be used to model local blocks of the human body, so as to obtain better human retrieval features. In a specific example, 1*1, 1*3, 3*1, 3*3, 3*5, 5*3 and 5*5 convolution kernels can be used in each first local perception branch, respectively, so as to perform multi-scale and multi-directional feature extraction.

[0087] Continue to see Figure 3D In an optional embodiment, a first bypass branch may be further added to the first local perception module to further accelerate model convergence. Optionally, the first bypass branch may be a direct connection structure, so that the first sample feature to be processed is directly used as the corresponding first local intermediate sample feature to reduce gradient dissipation, thereby accelerating model convergence. Or optionally, a first batch processing module may be set in the first bypass branch, corresponding to the batch processing layer, to normalize the first sample feature to be processed to obtain the corresponding first local intermediate sample feature, thereby assimilating the distribution of the first local intermediate sample features in different branches, thereby accelerating model convergence.

[0088] The above content provides an exemplary description of the specific structure of the feature extraction network in the joint perception model. The following will describe the joint perception network in the joint perception model in detail.

[0089] See also Figure 4A The schematic diagram of the joint perception network shown in the figure may include a detection head module, in which different task perception branches are arranged, which are used to jointly perceive the target sample features through different task perception branches to obtain the perception prediction results of the corresponding perception tasks.

[0090] Exemplarily, the task perception branch may include at least one of a target detection branch, an attribute recognition branch, and a search feature extraction branch, etc. Among them, the target detection branch corresponds to the target detection task, which is used to detect the object to be labeled in the image (that is, a certain category of target); the attribute recognition branch corresponds to the attribute recognition task, which is used to identify at least one attribute category of the object to be labeled in the image; the search feature extraction branch corresponds to the search feature extraction task, which is used to extract feature vectors that facilitate image search.

[0091] It should be noted that the above-mentioned different task perception branches are only illustrative of the different task perception branches in the detection head module and should not be understood as a specific limitation on the task perception branches. The task perception branches can be added, deleted or modified according to actual needs.

[0092] It can be understood that by setting up a detection head module with different task perception branches in the joint perception network, the joint perception network has the ability to jointly perceive different perception tasks. There is no need to establish and train a single perception model for a certain perception task separately, which improves the convenience in multi-perception tasks and the versatility of the joint perception model.

[0093] In order to enable the joint perception model to learn the joint perception capability of different perception tasks in different categories of targets, in an optional embodiment, the number of detection head modules in the joint perception network can also be set to at least two, and different detection head modules perform joint perception on different categories of targets. The advantage of this setting is that there is no need to train corresponding joint perception models for different categories of targets separately, while improving the versatility of the joint perception model. In addition, by sharing the same feature extraction network for different perception tasks under different categories of targets, a large amount of calculations can be saved. During the model training process, the feature extraction network can also learn the visual features of different perception tasks under different categories of targets, further improving the richness of the extracted features and helping to improve the model accuracy of the joint perception model.

[0094] In a specific example, a face detection head module, a human body detection head module, a motor vehicle detection head module, and a non-motor vehicle detection head module can be respectively set in the joint perception network. Among them, the face detection head module is used to perform joint perception of different perception tasks on face targets; the human body detection head module is used to perform joint perception of different perception tasks on human targets; the motor vehicle detection head module is used to perform joint perception of different perception tasks on motor vehicle targets; and the non-motor vehicle detection head module is used to perform joint perception of different perception tasks on non-motor vehicle targets.

[0095] When there are at least two types of targets in the sample image, in an optional embodiment, other network structures can be introduced to enhance the detection head module's targeting of different types of targets during the joint perception process.

[0096] See also Figure 4B The schematic diagram of the detection head module shown in FIG. The detection head module in the joint perception network may include a second multi-path module and at least two task perception branches (eg, the target detection branch, the attribute recognition branch, and the search feature extraction branch shown in the figure).

[0097] Among them, the second multi-channel module in the detection head module extracts the features of the target sample under different perception tasks of the same category of targets to obtain the perception sample features; each task perception branch determines the perception prediction result under the corresponding perception task based on the perception sample features.

[0098] That is, the second multi-channel module is used to extract the features of the target sample characteristics of the same category under different perception tasks to obtain the perception sample characteristics; each task perception branch shares the perception sample characteristics, and performs joint perception based on the perception sample characteristics to obtain the perception prediction results under the corresponding perception tasks.

[0099] It can be understood that by introducing a second multi-path module in the detection head module of the joint perception network, the target sample features output by the feature extraction network are used to extract visual features corresponding to different perception tasks under the same category of targets, so that the extracted features can be matched with the category targets corresponding to the detection head module, making the extracted visual features more target-specific, which helps to improve the accuracy of the perception prediction results of different perception tasks corresponding to different categories of targets.

[0100] It should be noted that the target sample features of a single scale can be used as the input data of each detection head module corresponding to the same category of targets, and the detection head modules can be jointly perceived. The joint perception results of the detection head modules for the same category of targets at different scales are fused according to the perception task dimension to obtain the final perception prediction results of the corresponding category of targets. Or, optionally, the target sample features of different scales can be fused, and the fusion results can be used as the input data of the detection head module to perform joint perception of the corresponding category of targets of the detection head modules to obtain the final perception prediction results of the corresponding category of targets.

[0101] The specific structure of the second multi-channel module in the detection head module of the joint perception network is described in detail below. It should be noted that the second multi-channel module in the detection head module has the same structure as the first multi-channel module in the aforementioned feature extraction network, and the network parameters can be the same or different, which is determined according to the actual training situation.

[0102] See also Figure 4C The structural schematic diagram of the second multi-channel module shown in the figure, the second multi-channel module can be further refined to include a second splitting subnetwork, a second feature extraction subnetwork and a second fusion subnetwork; the second splitting subnetwork splits the target sample features according to the channel dimension to obtain the second sample features to be fused and the second sample features to be processed; the second feature extraction subnetwork extracts features from the second sample features to be processed to obtain the second target intermediate sample features; the second fusion subnetwork fuses the second sample features to be fused and the second target intermediate sample features to obtain the perceived sample features.

[0103] Exemplarily, by splitting the target sample features according to the channel dimension, at least two groups of split results can be obtained. When splitting, the splitting can be performed randomly or sequentially according to the channel dimension. The present disclosure does not impose any restrictions on the specific splitting mechanism, and only needs to ensure that the splitting is performed according to the channel dimension. It should be noted that the present disclosure does not impose any restrictions on the number of channels of features in different splitting results.

[0104] In order to facilitate subsequent processing, the number of split results can be limited to two, that is, the target sample feature is split to obtain the second sample feature to be fused and the second sample feature to be processed. In order to facilitate the splitting operation, the target sample feature can be split into the second sample feature to be fused and the second sample feature to be processed by sequential splitting. Optionally, the number of channels of the second sample feature to be fused and the second sample feature to be processed can be the same.

[0105] It is understandable that the method of extracting features from the second sample features to be processed in the target sample features instead of extracting features from the full amount of target sample features reduces the number of channels in the feature extraction process (when the number of channels of the second sample features to be fused and the second sample features to be processed are the same, the number of channels is halved), thereby reducing the amount of computation and memory usage in the feature extraction process. At the same time, by fusing the second target intermediate sample features obtained by extracting the second sample features to be fused in the target sample features and the second sample features to be processed, the number of gradient paths is increased (when the number of channels of the second sample features to be fused and the second sample features to be processed are the same, the number of gradient paths is doubled), thereby improving the learning ability of the trained model.

[0106] Taking the target sample feature of shape (n,c,h,w) as an example, the processing process of the second multi-channel module is described. Where n is the number of sample images, c is the number of channels, h is the height of the sample image, and w is the width of the sample image. The target sample feature is evenly split along the channel dimension to obtain the second sample feature to be fused and the second sample feature to be processed, both of which have the shape of (n,c / 2,h,w). The second feature extraction subnetwork extracts the second sample feature to be processed of shape (n,c / 2,h,w) to obtain the second target intermediate sample feature of shape (n,c / 2,h,w). Since the number of channels in the feature extraction process is halved, the computational amount and video memory usage of the feature extraction process are reduced. By fusion of the second sample feature to be fused of shape (n,c / 2,h,w) and the second target intermediate sample feature of shape (n,c / 2,h,w), the target sample feature of shape (n,c,h,w) is obtained, and the number of gradient paths is doubled, thereby improving the learning ability of the trained model.

[0107] In order to further improve the comprehensiveness and richness of the features extracted by the second feature extraction sub-network, in an optional embodiment, features of different dimensions may be extracted in the second feature extraction sub-network.

[0108] See also Figure 4D The schematic diagram of the second feature extraction subnetwork shown in the figure can be configured to include a second global perception module, a second local perception module and a second perception fusion module; the second global perception module performs global feature extraction on the second sample features to be processed to obtain second global intermediate sample features; the second local perception module performs local feature extraction on the second sample features to be processed to obtain second local intermediate sample features; the second perception fusion module performs feature fusion on the second global intermediate sample features and the second local intermediate sample features to obtain second target intermediate sample features.

[0109] It can be understood that by extracting features of the second sample to be processed from the global dimension and the local dimension respectively, and fusing the obtained second global intermediate sample features and the second local intermediate sample features, the second target intermediate sample features obtained after the fusion carry both overall context information and local details, thereby improving the richness and comprehensiveness of the second target intermediate sample features.

[0110] In an alternative embodiment, see Figure 4E The schematic diagram of the second feature extraction subnetwork shown in the figure, the second global perception module of the second feature extraction subnetwork may include a pooling layer and an activation layer, wherein the pooling layer is used to perform global feature extraction on the second sample features to be processed; the activation layer is used to perform nonlinear calculation on the global feature extraction results, thereby improving the nonlinear feature extraction capability of the second global perception module.

[0111] It should be noted that the embodiments of the present disclosure do not limit the specific pooling method used by the pooling layer and the specific activation function used by the activation layer, and can be set or adjusted according to actual needs and model training conditions. For example, the pooling layer can use average pooling, and the activation layer can use ReLU activation function.

[0112] In an optional embodiment, at least one second local perception branch may be provided in the second local perception module according to actual needs to perform local feature extraction.

[0113] For example, see Figure 4EThe schematic diagram of the second feature extraction subnetwork shown in the figure, at least two second local perception branches can be set in the second local perception module of the second feature extraction subnetwork, and the second local perception branches are used to extract local features of the second sample features to be processed under different receptive fields to obtain the second local intermediate sample features under the corresponding receptive fields. Correspondingly, the second perception fusion module can first fuse the second local intermediate sample features under different receptive fields (for example, pixel superposition) to obtain the second local intermediate sample feature fusion result; fuse the second local intermediate sample feature fusion result with the second global intermediate sample feature (for example, channel multiplication) to obtain the second target intermediate sample feature.

[0114] It can be understood that by setting at least two second local perception branches to extract local features under different receptive fields, the richness of the second local intermediate sample features is improved, which helps to improve the labeling ability of objects to be labeled of different scales, and lays the foundation for the labeling of multi-category targets. At the same time, by extracting local features under different receptive fields, the extracted features can support different perception tasks, which helps to improve the perception accuracy of the joint perception model.

[0115] In a specific implementation, a convolution layer and a batch processing layer can be set in the second local perception branch. The convolution layer is used to extract local features under the corresponding receptive field according to the convolution kernel; the batch processing layer is used to normalize the extracted features, so as to assimilate the distribution of the second local intermediate sample features extracted by different second local perception branches, accelerate the convergence of the model, and improve the stability of the training process.

[0116] It should be noted that the convolution kernel scales used in different second local perception branches are different, so that different second local perception branches can perform local feature extraction under different receptive fields. Among them, the number of convolution kernels can be set or adjusted by technicians according to needs or experience values, and the type of convolution kernel can be specifically set according to the perception task of joint perception. Exemplarily, for target search feature extraction tasks, since the search process usually requires multi-granularity features, both large-scale convolution kernels and small-scale convolution kernels need to be set. Exemplarily, for attribute recognition tasks and target detection tasks, horizontal or vertical convolution kernels need to be set for different target categories. Specifically, 1*3 convolution kernels and 3*5 convolution kernels can be used to model local blocks of the human body, so as to obtain better human retrieval features. In a specific example, convolution kernels of 1*1, 1*3, 3*1, 3*3, 3*5, 5*3 and 5*5 can be used in each local perception branch, respectively, so as to perform multi-scale and multi-directional feature extraction.

[0117] Continue to see Figure 4EIn an optional embodiment, a second bypass branch may be further added to the second local perception module to further accelerate model convergence. Optionally, the second bypass branch may be a direct connection structure, so that the second sample feature to be processed is directly used as the corresponding second local intermediate sample feature to reduce gradient dissipation, thereby accelerating model convergence. Alternatively, a second batch processing module may be provided in the second bypass branch, corresponding to the batch processing layer, to normalize the second sample feature to be processed to obtain the corresponding second local intermediate sample feature, thereby assimilating the distribution of the second local intermediate sample features in different branches, thereby accelerating model convergence.

[0118] The above content describes in detail the training process of the joint perception model. The following will describe in detail the joint perception method through the use process of the joint perception model.

[0119] Figure 5 It is a flow chart of a joint perception method provided by an embodiment of the present disclosure, which is applicable to application scenarios in which a joint perception model with a multi-task joint perception function is used to perform joint perception of different perception tasks. The joint perception method provided by an embodiment of the present disclosure can be executed by a joint perception device, which can be implemented in software and / or hardware and specifically configured in an electronic device. It should be noted that the electronic device that executes the joint perception method and the electronic device that executes the joint perception model training method can be the same device or different devices, and the present disclosure does not impose any limitation on this.

[0120] See also Figure 5 A joint sensing method is shown, comprising:

[0121] S501: Obtain a predicted image.

[0122] The predicted image is an image to be jointly sensed. Exemplarily, the predicted image may be a picture frame obtained by performing frame extraction processing on the video stream data.

[0123] In an optional embodiment, in order to improve the accuracy of the subsequent perception prediction results of the prediction image, in an optional embodiment, the prediction image may also be preprocessed, wherein the preprocessing method is consistent with the preprocessing method of the sample image when training the joint perception model.

[0124] Exemplarily, the predicted image may be scaled to make the predicted image consistent with the size of the preprocessed sample image. Exemplarily, the predicted image may be normalized, for example, by subtracting the mean of the three primary colors RGB from the predicted image.

[0125] S502: Input the predicted image into the target joint perception model to obtain a perception prediction result.

[0126] The target joint perception model is obtained based on an initial joint perception model obtained by any joint perception model training method provided by the embodiments of the present disclosure. The perception prediction results may be prediction results corresponding to different perception tasks. For example, the perception prediction results may include at least one of the predicted annotation box position of the object to be annotated under the target detection task, at least one attribute category of the object to be annotated under the attribute recognition task, and the predicted feature vector for image search under the search feature extraction task.

[0127] Optionally, the previously trained joint perception model, that is, the initial joint perception model, can be directly used as the target joint perception model. Alternatively, the initial joint perception model can be post-processed to improve the reasoning performance, and the post-processing result can be used as the target joint perception model.

[0128] It should be noted that the target joint perception model adopted in the present invention is determined based on the aforementioned trained initial joint perception model, and the joint perception network in the initial joint perception model shares the target prediction features extracted by the feature extraction network for different perception tasks. There is no need to set up independent feature extraction networks for different perception tasks, which saves a lot of computing resources and improves computing efficiency.

[0129] When the feature extraction network of the initial joint perception model includes a first multi-path module, and the first multi-path module includes a first feature extraction sub-network, and the first local perception module in the first feature extraction sub-network is set to include at least two first local perception branches, due to the multiple local perception branches, in the reasoning stage, that is, directly using the trained initial joint perception model to determine the perceptual prediction result of the predicted image, a large amount of computing resources will be invested and the reasoning efficiency will be relatively low.

[0130] In order to reduce the amount of computation in the reasoning stage and improve the reasoning efficiency, the first local perception module in the initial joint perception model can be replaced by an equivalent module that consumes less time and requires less computation to obtain the target joint perception model.

[0131] In an optional embodiment, the target joint perception model can be obtained based on re-parameterization of at least two first local perception branches. That is, at least two first local perception branches in the first local perception module of the first feature extraction subnetwork in the first multi-channel module in the feature extraction network in the initial joint perception model can be re-parameterized, and the single-channel structure obtained after the re-parameterization can replace the original at least two first local perception branches. Alternatively, if the first local perception module of the first multi-channel module includes at least two first local perception branches and a first bypass branch, at least two first local perception branches and a first bypass branch in the first local perception module of the first multi-channel module in the feature extraction network in the initial joint perception model can be re-parameterized together, and the single-channel structure obtained after the re-parameterization can replace the original at least two first local perception branches and the first bypass branch.

[0132] It should be noted that the computing device used to perform reparameterization can be an electronic device that performs the joint perception model training method, an electronic device that performs the joint perception method, or other devices, and the present disclosure does not impose any restrictions on this. In addition, the present disclosure does not impose any restrictions on the specific execution time of reparameterization, and it only needs to ensure that it is performed after the initial joint perception model is trained and before the target joint perception model is used.

[0133] It can be understood that by reparameterizing the first multi-way module of the feature extraction network in the initial joint perception model, the complexity of the feature extraction network in the joint perception model used can be reduced while maintaining rich feature extraction, thereby reducing the amount of data calculation in the reasoning process in which the first feature extraction subnetwork processes the input data to obtain the corresponding output results, thereby improving the reasoning efficiency.

[0134] In a specific implementation, the parameters of the convolutional layer and the batch processing layer in the first local perception branch in the feature extraction network can be first merged to obtain the parameter merging results of different first local perception branches; then the network parameters of different first local perception branches and / or first bypass branches are merged to obtain the final re-parameterization result; the final parameterization result is used to optimize the equivalent alternative structural setting of the first local perception module in the first feature extraction subnetwork in the first multi-path module in the feature extraction network.

[0135] Specifically, the following formula can be used to reparameterize the first local perception module:

[0136]

[0137]

[0138] Among them, F i is the convolution kernel of the i-th branch convolution layer, μi and σ i is the mean and variance of the batch layer of the ith branch, γ i and β i are the scale factor and translation factor of the batch layer. F' and b' are the parameters of the reparameterized convolution kernel and bias term.

[0139] See also Fig. 6A The schematic diagram of the first feature extraction subnetwork before and after re-parameterization shown in FIG. 1 is a schematic diagram of the first feature extraction subnetwork before and after re-parameterization, wherein the first local perception module in the first feature extraction subnetwork replaces the multi-branch first local perception module in the initial joint perception model (see Figure 3D ) is replaced with a single-path structure, and the convolution layer is configured only according to the re-parameterized result. The model result is greatly simplified, which helps to improve the reasoning efficiency. In a specific example, if the convolution kernels of 1*1, 1*3, 3*1, 3*3, 3*5, 5*3 and 5*5 are used in each first local perception branch of the first local perception module in the model training stage, the convolution kernel size of the re-parameterized convolution layer is 5*5.

[0140] When the joint perception network of the initial joint perception model includes a detection head module, and a second multi-way module is arranged in the detection head module; the second multi-way module includes a second feature extraction sub-network, and the second local perception module in the second feature extraction sub-network is arranged to include at least two second local perception branches, due to the method of multiple local perception branches, in the reasoning stage, that is, directly using the trained initial joint perception model to determine the perceptual prediction result of the predicted image, a large amount of computing resources will be invested, and the reasoning efficiency will be relatively low.

[0141] In order to reduce the amount of computation in the reasoning stage and improve the reasoning efficiency, the second local perception module of the second multi-path module in the initial joint perception model can be replaced by an equivalent module that consumes less time and requires less computation to obtain the target joint perception model.

[0142] In an optional embodiment, the target joint perception model can be obtained based on re-parameterizing at least two second local perception branches. That is, at least two second local perception branches in the second local perception module of the second multi-channel module in the detection head module in the initial joint perception model can be re-parameterized, and the single-channel structure obtained after the re-parameterization can replace the original at least two second local perception branches of the second local perception module in the second multi-channel module of the detection head module. Alternatively, if the second local perception module of the second multi-channel module includes a second local perception branch and a second bypass branch, at least two second local perception branches and a second bypass branch in the second local perception module of the second multi-channel module in the detection head module in the initial joint perception model can be re-parameterized together, and the single-channel structure obtained after the re-parameterization can replace the original at least two second local perception branches and the second bypass branch.

[0143] It should be noted that the computing device used to perform reparameterization can be an electronic device that performs the joint perception model training method, an electronic device that performs the joint perception method, or other devices, and the present disclosure does not impose any restrictions on this. In addition, the present disclosure does not impose any restrictions on the specific execution time of reparameterization, and it only needs to ensure that it is performed after the initial joint perception model is trained and before the target joint perception model is used.

[0144] It can be understood that by reparameterizing the second multi-way module of the joint perception network in the initial joint perception model, the complexity of the joint perception network in the joint perception model used can be reduced while maintaining rich feature extraction, thereby reducing the amount of data calculation in the reasoning process in which the second feature extraction subnetwork processes the input data to obtain the corresponding output results, thereby improving the reasoning efficiency.

[0145] In a specific implementation, the parameters of the convolutional layer and batch processing layer in the second local perception branch in the detection head module of the joint perception network can be first merged to obtain the parameter merging results of different second local perception branches; then the network parameters of different second local perception branches and / or second bypass branches are merged to obtain the final re-parameterization result; the final re-parameterization result is used to perform an equivalent replacement structure setting of the second local perception module in the second feature extraction subnetwork in the second multi-path module in the joint perception network.

[0146] Specifically, the second local perception module can be reparameterized using the following formula:

[0147]

[0148]

[0149] Among them, F iis the convolution kernel of the i-th branch convolution layer, μ i and σ i is the mean and variance of the batch layer of the ith branch, γ i and β i are the scale factor and translation factor of the batch layer. F' and b' are the parameters of the reparameterized convolution kernel and bias term.

[0150] See also Figure 6B The schematic diagram of the second feature extraction subnetwork before and after re-parameterization shown in FIG. 1 is a schematic diagram of the second feature extraction subnetwork before and after re-parameterization, wherein the second local perception module in the second feature extraction subnetwork replaces the multi-branch second local perception module in the initial joint perception model (see Figure 4E ) is replaced with a single-path structure, and the convolution layer is configured only according to the re-parameterized result. The model result is greatly simplified, which helps to improve the reasoning efficiency. In a specific example, if the convolution kernels of 1*1, 1*3, 3*1, 3*3, 3*5, 5*3 and 5*5 are used in each second local perception branch of the second local perception module in the model training stage, the convolution kernel size of the re-parameterized convolution layer is 5*5.

[0151] In an optional embodiment, on a device that supports shaping quantization (such as int8) inference, shaping quantization can also be performed on the initial joint perception model or the re-parameterized initial joint perception model, thereby further improving the inference speed.

[0152] When the first bypass branch of the first local perception module in the first multi-channel module includes the first batch of processing modules, the scale variance within the first batch of processing modules is large, resulting in uneven distribution of convolutional layer weights in the equivalent replacement structure after reparameterization, and poor effect of shaping quantization, which will affect the accuracy of the reasoning result. In order to avoid the above situation, in an optional embodiment, before reparameterizing the first local perception module of the first multi-channel module, the first bypass branch including the first batch of processing modules can also be replaced with a direct connection structure, which does not affect the overall model effect and is also beneficial to the distribution of convolutional layer weights after reparameterization.

[0153] When the second bypass branch of the second local perception module in the second multi-channel module includes the second batch processing module, the scale variance within the second batch processing module is large, resulting in uneven distribution of convolutional layer weights in the equivalent replacement structure after reparameterization, and poor effect of shaping quantization, which will affect the accuracy of the reasoning result. In order to avoid the above situation, in an optional embodiment, before reparameterizing the second local perception module of the second multi-channel module, the second bypass branch including the second batch processing module can also be replaced with a direct connection structure, which does not affect the overall model effect and is also beneficial to the distribution of convolutional layer weights after reparameterization.

[0154] As an implementation of the above-mentioned joint perception model training methods, the present disclosure also provides an optional embodiment of an execution device for implementing the above-mentioned joint perception model training methods. Figure 7 The joint perception model training device 700 shown includes: a sample data acquisition module 701, a preset model acquisition module 702, a feature extraction network 703, a joint perception network 704 and a model training module 705. The sample data acquisition module 701 is used to acquire sample images and perception labels of sample images;

[0155] The preset model acquisition module 702 is used to acquire a preset joint perception model; the joint perception model includes a feature extraction network and a joint perception network;

[0156] A feature extraction network 703 is used to extract features from the sample image to obtain target sample features;

[0157] A joint sensing network 704 is used to perform joint sensing according to the target sample characteristics to obtain a sensing prediction result;

[0158] The model training module 705 is used to train the pre-built joint perception model according to the perception prediction results and the perception labels; wherein the joint perception includes performing at least two perception tasks.

[0159] In an optional embodiment, the feature extraction network 703 includes a base module and at least two first multiplexer modules;

[0160] The base module is used to downsample the sample image to obtain initial sample features of different scales;

[0161] The first multi-channel module is used to extract features of different perception tasks from the initial sample features to obtain target sample features at a corresponding scale of the initial sample features.

[0162] In an optional embodiment, the first multi-channel module includes a first splitting sub-network, a first feature extraction sub-network and a first fusion sub-network;

[0163] A first splitting subnetwork is used to split the initial sample features according to the channel dimension to obtain a first sample feature to be fused and a first sample feature to be processed;

[0164] A first feature extraction subnetwork is used to extract features of the first sample to be processed to obtain first target intermediate sample features;

[0165] The first fusion sub-network is used to perform feature fusion on the first sample feature to be fused and the first target intermediate sample feature to obtain the target sample feature at a corresponding scale of the initial sample feature.

[0166] In an optional embodiment, the first feature extraction subnetwork includes a first global perception module, a first local perception module and a first perception fusion module;

[0167] A first global perception module is used to perform global feature extraction on the first sample feature to be processed to obtain a first global intermediate sample feature;

[0168] A first local perception module is used to extract local features of the first sample feature to be processed to obtain a first local intermediate sample feature;

[0169] The first perception fusion module is used to perform feature fusion on the first global intermediate sample feature and the first local intermediate sample feature to obtain a first target intermediate sample feature.

[0170] In an optional embodiment, the first local perception module includes at least two first local perception branches;

[0171] Each first local perception branch is used to perform local feature extraction on the first sample feature to be processed under different receptive fields to obtain the first local intermediate sample feature under the corresponding receptive field.

[0172] In an optional embodiment, the first local perception module further includes a first bypass branch;

[0173] If the first bypass branch is a direct connection structure, the first bypass branch is used to directly use the first to-be-processed sample feature as the corresponding first local intermediate sample feature; or,

[0174] If the first bypass branch includes the first batch of processing modules, the first bypass branch is used to perform normalization processing on the first to-be-processed sample features through the first batch of processing modules to obtain corresponding first local intermediate sample features.

[0175] In an optional embodiment, the joint sensing network 704 includes a detection head module;

[0176] The detection head module is used to jointly perceive the target sample features through different task perception branches to obtain the perception prediction results of the corresponding perception tasks.

[0177] In an optional embodiment, the detection head module includes a second multiplexer module and at least two task sensing branches;

[0178] The second multi-channel module is used to extract the features of the target sample under different perception tasks of the same category of targets to obtain the perception sample features;

[0179] Each task perception branch is used to determine the perception prediction result under the corresponding perception task according to the perception sample characteristics.

[0180] In an optional embodiment, the second multi-channel module includes a second splitting sub-network, a second feature extraction sub-network, and a second fusion sub-network;

[0181] The second splitting subnetwork is used to split the target sample features according to the channel dimension to obtain the second sample features to be fused and the second sample features to be processed;

[0182] A second feature extraction subnetwork is used to extract features of the second sample to be processed to obtain second target intermediate sample features;

[0183] The second fusion sub-network is used to fuse the second sample features to be fused and the second target intermediate sample features to obtain the perceived sample features.

[0184] In an optional embodiment, the second feature extraction subnetwork includes a second global perception module, a second local perception module, and a second perception fusion module;

[0185] A second global perception module is used to perform global feature extraction on the second sample feature to be processed to obtain a second global intermediate sample feature;

[0186] A second local perception module is used to extract local features of the second sample feature to be processed to obtain a second local intermediate sample feature;

[0187] The second perception fusion module is used to perform feature fusion on the second global intermediate sample feature and the second local intermediate sample feature to obtain a second target intermediate sample feature.

[0188] In an optional embodiment, the second local perception module includes at least two second local perception branches;

[0189] Each second local perception branch is used to perform local feature extraction on the second sample feature to be processed under different receptive fields to obtain the second local intermediate sample feature under the corresponding receptive field.

[0190] In an optional embodiment, the second local perception module further includes a second bypass branch;

[0191] If the second bypass branch is a direct connection structure, the second bypass branch is used to directly use the second to-be-processed sample feature as the corresponding second local intermediate sample feature; or,

[0192] If the second bypass branch includes a second batch processing module, the second bypass branch is used to perform normalization processing on the second to-be-processed sample features through the second batch processing module to obtain corresponding second local intermediate sample features.

[0193] In an optional embodiment, the number of detection head modules is at least two; different detection head modules jointly perceive targets of different categories.

[0194] The above-mentioned joint perception model training device can execute the joint perception model training method provided by any embodiment of the present disclosure, and has corresponding functional modules and beneficial effects for executing each joint perception model training method.

[0195] As an implementation of the above-mentioned joint perception methods, the present disclosure also provides an optional embodiment of an execution device for implementing the above-mentioned joint perception methods. Figure 8 The joint perception device 800 shown includes: a predicted image acquisition module 801 and a perception prediction module 802.

[0196] The predicted image acquisition module 801 is used to acquire the predicted image;

[0197] The perception prediction module 802 is used to input the predicted image into the target joint perception model to obtain a perception prediction result; wherein the target joint perception model is obtained based on an initial joint perception model trained by any one of the joint perception model training devices provided in the embodiments of the present disclosure.

[0198] In an optional embodiment, the target joint perception model is obtained based on shaping and quantizing the network parameters of the initial joint perception model.

[0199] In an optional embodiment, the initial joint perception model includes a first multi-path module, the first multi-path module includes at least two first local perception branches; the target joint perception model is obtained based on re-parameterization of the at least two first local perception branches; and / or,

[0200] The initial joint perception model includes a second multi-path module, and the second multi-path module includes at least two second local perception branches; the target joint perception model is obtained based on re-parameterization of the at least two second local perception branches.

[0201] The above-mentioned joint perception device can execute the joint perception method provided by any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects for executing each joint perception method.

[0202] In the technical solution disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of sample images, perception labels and predicted images involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0203] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0204] Fig. 9A schematic block diagram of an example electronic device 900 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0205] like Fig. 9 As shown, the device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0206] A number of components in the device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0207] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above, such as the joint perception model training method and / or the joint perception method. For example, in some embodiments, the joint perception model training method and / or the joint perception method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the joint perception model training method and / or the joint perception method described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to perform the joint perception model training method and / or the joint perception method in any other suitable manner (e.g., by means of firmware).

[0208] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0209] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0210] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0211] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0212] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0213] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services. The server may also be a server of a distributed system, or a server combined with a blockchain.

[0214] Artificial intelligence is a discipline that studies how to use computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.). It includes both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing; artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, machine learning / deep learning technology, big data processing technology, knowledge graph technology, and other major directions.

[0215] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions provided by this disclosure can be achieved, and this document does not limit this.

[0216] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A joint perception model training method, comprising: Obtaining a sample image and a perceptual label of the sample image; Acquire a preset joint perception model; the joint perception model includes a feature extraction network and a joint perception network; Extracting features from the sample image through the feature extraction network to obtain target sample features; Performing joint perception according to the target sample characteristics through the joint perception network to obtain a perception prediction result; Training a preset joint perception model according to the perception prediction result and the perception label; wherein the joint perception includes performing at least two perception tasks; Wherein, the joint perception network includes a detection head module; the detection head module includes a second multi-channel module and at least two task perception branches; wherein the second multi-channel module includes a second splitting sub-network, a second feature extraction sub-network and a second fusion sub-network; The performing joint perception according to the target sample features through the joint perception network to obtain a perception prediction result includes: Splitting the target sample features according to the channel dimension through the second splitting sub-network to obtain a second sample feature to be fused and a second sample feature to be processed; Extracting features of the second sample to be processed by the second feature extraction subnetwork to obtain second target intermediate sample features; The second to-be-fused sample feature and the second target intermediate sample feature are subjected to feature fusion through the second fusion sub-network to obtain a perception sample feature; The perception prediction results under the corresponding perception tasks are determined by each task perception branch according to the perception sample characteristics.

2. The method according to claim 1, wherein: The feature extraction network includes a base module and at least two first multi-channel modules; The step of extracting features from the sample image through the feature extraction network to obtain target sample features includes: Downsampling the sample image through the base module to obtain initial sample features of different scales; The first multi-channel module performs feature extraction of different perception tasks on the initial sample features to obtain target sample features at a corresponding scale of the initial sample features.

3. The method according to claim 2, wherein: The first multi-channel module includes a first splitting sub-network, a first feature extraction sub-network and a first fusion sub-network; The first multi-channel module performs feature extraction of different perception tasks on the initial sample features to obtain target sample features at a corresponding scale of the initial sample features, including: The initial sample features are randomly split, sequentially split or evenly split according to the channel dimension by the first splitting sub-network to obtain first sample features to be fused and first sample features to be processed; Extracting features of the first sample to be processed by the first feature extraction subnetwork to obtain first target intermediate sample features; The first sample feature to be fused and the first target intermediate sample feature are fused through the first fusion sub-network to obtain the target sample feature at a corresponding scale of the initial sample feature.

4. The method according to claim 3, wherein: The first feature extraction subnetwork includes a first global perception module, a first local perception module and a first perception fusion module; The extracting the first to-be-processed sample feature by the first feature extraction sub-network to obtain the first target intermediate sample feature includes: Performing global feature extraction on the first sample feature to be processed by the first global perception module to obtain a first global intermediate sample feature; Performing local feature extraction on the first to-be-processed sample feature by the first local perception module to obtain a first local intermediate sample feature; The first global intermediate sample feature and the first local intermediate sample feature are fused through the first perception fusion module to obtain the first target intermediate sample feature.

5. The method according to claim 4, wherein: The first local perception module includes at least two first local perception branches; The extracting local features of the first sample feature to be processed by the first local perception module to obtain a first local intermediate sample feature includes: Local feature extraction is performed on the first sample feature to be processed in different receptive fields through different first local perception branches to obtain first local intermediate sample features in corresponding receptive fields.

6. The method according to claim 5, wherein: The first local perception module also includes a first bypass branch; The extracting local features of the first sample feature to be processed by the first local perception module to obtain a first local intermediate sample feature also includes: If the first bypass branch is a direct connection structure, the first to-be-processed sample feature is directly used as the corresponding first local intermediate sample feature; or, If the first bypass branch includes a first batch of processing modules, the first batch of processing modules are used to perform normalization processing on the first to-be-processed sample features to obtain corresponding first local intermediate sample features.

7. The method according to claim 1, wherein: The second feature extraction subnetwork includes a second global perception module, a second local perception module and a second perception fusion module; The extracting the second to-be-processed sample features by the second feature extraction sub-network to obtain the second target intermediate sample features includes: Performing global feature extraction on the second sample feature to be processed by the second global perception module to obtain a second global intermediate sample feature; Performing local feature extraction on the second sample feature to be processed by the second local perception module to obtain a second local intermediate sample feature; The second global intermediate sample feature and the second local intermediate sample feature are subjected to feature fusion through the second perception fusion module to obtain the second target intermediate sample feature.

8. The method according to claim 7, wherein: The second local perception module includes at least two second local perception branches; The extracting local features of the second sample feature to be processed by the second local perception module to obtain a second local intermediate sample feature includes: Local feature extraction is performed on the second sample feature to be processed in different receptive fields through different second local perception branches to obtain second local intermediate sample features in corresponding receptive fields.

9. The method according to claim 8, wherein: The second local perception module also includes a second bypass branch; The extracting local features of the second sample feature to be processed by the second local perception module to obtain a second local intermediate sample feature also includes: If the second bypass branch is a direct connection structure, the second to-be-processed sample feature is directly used as the corresponding second local intermediate sample feature; or, If the second bypass branch includes a second batch processing module, the second batch processing module is used to normalize the second to-be-processed sample features to obtain corresponding second local intermediate sample features.

10. The method according to claim 1, wherein: The number of the detection head modules is at least two; different detection head modules jointly perceive targets of different categories.

11. A joint sensing method, comprising: Get the predicted image; Inputting the predicted image into a target joint perception model to obtain a perception prediction result; The target joint perception model is obtained based on an initial joint perception model trained by the joint perception model training method according to any one of claims 1 to 10.

12. The method according to claim 11, wherein: The target joint perception model is obtained based on shaping and quantizing the network parameters of the initial joint perception model.

13. The method according to claim 11 or 12, wherein: The initial joint perception model includes a first multi-path module, the first multi-path module includes at least two first local perception branches; the target joint perception model is obtained based on re-parameterization of at least two first local perception branches; and / or, The initial joint perception model includes a second multi-path module, and the second multi-path module includes at least two second local perception branches; the target joint perception model is obtained based on re-parameterization of at least two second local perception branches.

14. A joint perception model training device, comprising: A sample data acquisition module, used to acquire a sample image and a perceptual label of the sample image; A preset model acquisition module, used to acquire a preset joint perception model; the joint perception model includes a feature extraction network and a joint perception network; A feature extraction network is used to extract features from the sample image to obtain target sample features; A joint perception network, used for performing joint perception according to the target sample characteristics to obtain a perception prediction result; A model training module, used for training a pre-built joint perception model according to the perception prediction result and the perception label; wherein the joint perception includes performing at least two perception tasks; Wherein, the joint perception network includes a detection head module; the detection head module includes a second multi-channel module and at least two task perception branches; wherein the second multi-channel module includes a second splitting sub-network, a second feature extraction sub-network and a second fusion sub-network; A second splitting subnetwork is used to split the target sample features according to the channel dimension to obtain a second sample feature to be fused and a second sample feature to be processed; A second feature extraction subnetwork is used to extract features of the second sample to be processed to obtain second target intermediate sample features; A second fusion sub-network is used to perform feature fusion on the second sample feature to be fused and the second target intermediate sample feature to obtain a perception sample feature; Each of the task perception branches is used to determine the perception prediction result under the corresponding perception task according to the perception sample characteristics.

15. The device according to claim 14, wherein: The feature extraction network includes a base module and at least two first multi-channel modules; The base module is used to downsample the sample image to obtain initial sample features of different scales; The first multi-channel module is used to perform feature extraction of different perception tasks on the initial sample features to obtain target sample features at a corresponding scale of the initial sample features.

16. The device according to claim 15, wherein: The first multi-channel module includes a first splitting sub-network, a first feature extraction sub-network and a first fusion sub-network; The first splitting subnetwork is used to randomly split, sequentially split or evenly split the initial sample features according to the channel dimension to obtain a first sample feature to be fused and a first sample feature to be processed; The first feature extraction subnetwork is used to extract features of the first sample to be processed to obtain first target intermediate sample features; The first fusion sub-network is used to perform feature fusion on the first sample feature to be fused and the first target intermediate sample feature to obtain the target sample feature at a corresponding scale of the initial sample feature.

17. The device according to claim 16, wherein: The first feature extraction subnetwork includes a first global perception module, a first local perception module and a first perception fusion module; The first global perception module is used to perform global feature extraction on the first sample feature to be processed to obtain a first global intermediate sample feature; The first local perception module is used to extract local features from the first sample feature to be processed to obtain a first local intermediate sample feature; The first perception fusion module is used to perform feature fusion on the first global intermediate sample feature and the first local intermediate sample feature to obtain the first target intermediate sample feature.

18. The device according to claim 17, wherein: The first local perception module includes at least two first local perception branches; Each of the first local perception branches is used to perform local feature extraction on the first sample feature to be processed under different receptive fields to obtain the first local intermediate sample feature under the corresponding receptive field.

19. The device according to claim 18, wherein: The first local perception module also includes a first bypass branch; If the first bypass branch is a direct connection structure, the first bypass branch is used to directly use the first to-be-processed sample feature as the corresponding first local intermediate sample feature; or, If the first bypass branch includes a first batch of processing modules, the first bypass branch is used to perform normalization processing on the first to-be-processed sample features through the first batch of processing modules to obtain corresponding first local intermediate sample features.

20. The device according to claim 14, wherein: The second feature extraction subnetwork includes a second global perception module, a second local perception module and a second perception fusion module; The second global perception module is used to perform global feature extraction on the second sample feature to be processed to obtain a second global intermediate sample feature; The second local perception module is used to extract local features from the second sample feature to be processed to obtain a second local intermediate sample feature; The second perception fusion module is used to perform feature fusion on the second global intermediate sample feature and the second local intermediate sample feature to obtain the second target intermediate sample feature.

21. The device according to claim 20, wherein: The second local perception module includes at least two second local perception branches; Each second local perception branch is used to perform local feature extraction on the second sample feature to be processed under different receptive fields to obtain a second local intermediate sample feature under the corresponding receptive field.

22. The device according to claim 21, wherein The second local perception module also includes a second bypass branch; If the second bypass branch is a direct connection structure, the second bypass branch is used to directly use the second to-be-processed sample feature as the corresponding second local intermediate sample feature; or, If the second bypass branch includes a second batch processing module, the second bypass branch is used to perform normalization processing on the second to-be-processed sample features through the second batch processing module to obtain corresponding second local intermediate sample features.

23. The device according to claim 14, wherein: The number of the detection head modules is at least two; different detection head modules jointly perceive targets of different categories.

24. A joint sensing device, comprising: A predicted image acquisition module, used for acquiring a predicted image; A perception prediction module is used to input the predicted image into a target joint perception model to obtain a perception prediction result; wherein the target joint perception model is obtained based on an initial joint perception model trained by the joint perception model training device described in any one of claims 14-23.

25. The device according to claim 24, wherein: The target joint perception model is obtained based on shaping and quantizing the network parameters of the initial joint perception model.

26. The device according to claim 24 or 25, wherein: The initial joint perception model includes a first multi-path module, the first multi-path module includes at least two first local perception branches; the target joint perception model is obtained based on re-parameterization of at least two first local perception branches; and / or, The initial joint perception model includes a second multi-path module, and the second multi-path module includes at least two second local perception branches; the target joint perception model is obtained based on re-parameterization of at least two second local perception branches.

27. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the joint perception model training method described in any one of claims 1-10, and / or the joint perception method described in any one of claims 11-13.

28. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to enable a computer to execute the joint perception model training method according to any one of claims 1-10, and / or the joint perception method according to any one of claims 11-13.

29. A computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the joint perception model training method described in any one of claims 1 to 10, and / or the steps of the joint perception method described in any one of claims 11 to 13.

Images