Model training method, target detection method, equipment and storage medium

Abstract

The invention discloses a model training method, a target detection method, equipment and a storage medium, and belongs to the technical field of computers. The method comprises the steps that a pre-training model is acquired, the pre-training model comprises a deep fusion network, and the deep fusion network is used for fusing language features and image features; a specified bypass network is added in the pre-training model to obtain a specified model, the specified bypass network comprises a bypass network of a specified layer in a fusion module in the deep fusion network, and the specified layer comprises at least one of a query layer, a key layer and a value layer in the fusion module. And training the specified model by using the training sample set to obtain a target detection model, in the training process, freezing all original weights in the pre-training model, and only adjusting the weight of the specified bypass network. According to the method, the weight needing to be adjusted in the model training process can be reduced, model training can be completed only through a small number of training samples, time and computing power needed by model training can be reduced, and the model training cost is reduced.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a model training method, a target detection method, a device and a storage medium. Background Art

[0002] Object detection is one of the important research topics in the field of computer vision and has a wide range of application values ​​in scenarios such as intelligent transportation and intelligent monitoring. Object detection refers to detecting objects from complex scene images so that they can be subsequently classified or tracked.

[0003] At present, various target detection requirements often arise in various scenarios. Each time a new requirement is proposed, a new target detection model needs to be trained to detect new targets. However, since target detection models are usually highly complex, training a new target detection model often requires a large number of data sets and takes a lot of time, which leads to high model training costs. Summary of the invention

[0004] The present application provides a model training method, target detection method, device and storage medium, which can reduce the cost of model training. The technical solution is as follows:

[0005] In a first aspect, a model training method is provided, the method comprising:

[0006] Obtain a pre-trained model, the pre-trained model is used to identify multiple targets, the pre-trained model includes a language backbone network, an image backbone network, and a deep fusion network, the language backbone network is used to extract language features, the image backbone network is used to extract image features, the deep fusion network is used to fuse the language features and the image features, and the deep fusion network includes a fusion module;

[0007] Adding a specified bypass network to the pre-trained model to obtain a specified model, wherein the specified bypass network includes a bypass network of a specified layer in the fusion module, and the specified layer includes at least one of a query layer, a key layer, and a value layer in the fusion module;

[0008] The designated model is trained using a training sample set to obtain a target detection model; wherein, during the training process, weights in the designated model except for weights of the designated bypass network are frozen.

[0009] In the present application, a pre-trained model is obtained, and then a specified bypass network is added to the pre-trained model to obtain a specified model. Afterwards, the specified model is trained using a training sample set to obtain a target detection model. Among them, all the original weights in the pre-trained model are frozen during the training process, and only the weights of the specified bypass network are adjusted. Since the fusion module in the pre-trained model is used to fuse language features and image features, and the specified bypass network includes at least one bypass network in the query layer, key layer, and value layer in the fusion module, the adjustment of the weight of the specified bypass network during the training process can enhance the semantic association between language features and image features, and a target detection model with a higher accuracy can be obtained. In addition, since only the weight of the specified bypass network needs to be adjusted during the training process, not only can the weights that need to be adjusted during the model training process be greatly reduced, but also only a small number of training samples are required to complete the model training. In this way, the time and computing power required for model training can be greatly reduced, thereby reducing the cost of model training.

[0010] Optionally, the designated bypass network also includes a bypass network of an input linear layer in the fusion module corresponding to the designated layer, and / or the designated bypass network also includes a bypass network of an output linear layer in the fusion module.

[0011] In the present application, the designated bypass network added to the pre-trained model includes a bypass network of at least one of the query layer, key layer, and value layer in the fusion module, and further includes a bypass network of the input linear layer and / or output linear layer in the fusion module. Therefore, when the pre-trained model already has a certain recognition ability for the preset target, the adjustment of the weight of the designated bypass network during the training process can strengthen the semantic association between the language features and the image features, and can also speed up the establishment of the matching relationship between the text and the image. In this way, only the weight of the designated bypass network is adjusted during the training process to obtain a target detection model that can accurately identify the preset target.

[0012] Optionally, adding a specified bypass network to the pre-trained model to obtain a specified model includes:

[0013] Determine all the designated layers in the pre-trained model according to the preset layer identifier;

[0014] A bypass network is constructed for each of the designated layers in the pre-trained model to obtain the designated model, wherein the bypass network includes a dimensionality reduction layer and a dimensionality increase layer, wherein the weight matrix of the dimensionality reduction layer obeys a Gaussian distribution after being initialized, and the weight matrix of the dimensionality increase layer is a zero matrix after being initialized.

[0015] In the present application, all designated layers in the pre-trained model can be simply and quickly located according to the preset layer identifier, and then the bypass network of each designated layer can be quickly constructed.

[0016] Optionally, before using the training sample set to train the specified model, the method further includes:

[0017] Receiving a plurality of images sent by a first device, wherein a first model is provided in the first device, the first model is obtained by training the pre-trained model or the specified model, and each of the plurality of images is an image containing a preset target recognized by the first device through the first model;

[0018] The training sample set is generated according to the multiple images, and each training sample in the training sample set includes an image sample, a text sample and a sample label.

[0019] In the present application, the first device can be used to automatically screen out multiple images containing preset targets. Compared with the method in the related art that requires technicians to manually screen images, the present application can greatly reduce labor costs.

[0020] Optionally, generating the training sample set according to the multiple images includes:

[0021] For any one of the multiple images, obtaining first annotation information of the image, where the first annotation information includes a target area position and a target category;

[0022] The one image is used as an image sample of a training sample, the target category is used as a text sample of the one training sample, and a sample label of the one training sample is determined according to the first annotation information to obtain the one training sample.

[0023] Optionally, generating the training sample set according to the multiple images includes:

[0024] For any one of the multiple images, obtaining first annotation information and text description information of the image, wherein the first annotation information includes a target region position and a target category, and the text description information is a description of the image;

[0025] Acquire second annotation information of the text description information, wherein the second annotation information includes a target word position and a target category;

[0026] The one image is used as an image sample of a training sample, the text description information is used as a text sample of the one training sample, and a sample label of the one training sample is determined according to the first annotation information and the second annotation information to obtain the one training sample.

[0027] Optionally, after the specified model is trained using the training sample set to obtain the target detection model, the method further includes:

[0028] Generate a weight file carrying the weights of the specified bypass network in the target detection model;

[0029] The weight file is sent to the second device, where the weight file is used to update the second model set in the second device.

[0030] In the present application, since the weight file only carries the weight of the specified bypass network in the target detection model, the file size of the weight file is relatively small. Since the file size of the weight file is relatively small, the speed of sending the weight file to the second device is relatively fast, and the speed of the second device updating the second model according to the weight file is also relatively fast, which can greatly improve the model deployment speed.

[0031] In a second aspect, a target detection method is provided, the method comprising:

[0032] Acquire an image to be detected;

[0033] Whether the image contains a preset target is determined according to a target detection model, and the target detection model is trained by the model training method described in the first aspect above.

[0034] In the present application, after acquiring the image to be detected, it is possible to determine whether the image contains a preset target based on the target detection model, thereby enabling fast and accurate target detection of the image.

[0035] In a third aspect, a model training device is provided, the device comprising:

[0036] An acquisition module is used to acquire a pre-trained model, wherein the pre-trained model is used to identify multiple targets, and the pre-trained model includes a language backbone network, an image backbone network, and a deep fusion network. The language backbone network is used to extract language features, the image backbone network is used to extract image features, the deep fusion network is used to fuse the language features and the image features, and the deep fusion network includes a fusion module;

[0037] An adding module, used for adding a specified bypass network to the pre-trained model to obtain a specified model, wherein the specified bypass network includes a bypass network of a specified layer in the fusion module, and the specified layer includes at least one of a query layer, a key layer, and a value layer in the fusion module;

[0038] A training module is used to train the specified model using a training sample set to obtain a target detection model; wherein, during the training process, the weights in the specified model except the weights of the specified bypass network are frozen.

[0039] In a fourth aspect, a target detection device is provided, the device comprising:

[0040] An acquisition module, used for acquiring an image to be detected;

[0041] A detection module is used to determine whether the image contains a preset target based on a target detection model, and the target detection model is trained by the model training method described in the first aspect above.

[0042] In a fifth aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the model training method described in the first aspect or the target detection method described in the second aspect is implemented.

[0043] In a sixth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the model training method described in the first aspect or the target detection method described in the second aspect is implemented.

[0044] In the seventh aspect, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute the steps of the model training method described in the first aspect or the steps of the target detection method described in the second aspect.

[0045] It can be understood that the beneficial effects of the second to seventh aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0047] Figure 1 is a flow chart of a model training method provided in an embodiment of the present application;

[0048] Figure 2 is a schematic diagram of a pre-training model provided in an embodiment of the present application;

[0049] Figure 3 is a schematic diagram of a fusion module provided in an embodiment of the present application;

[0050] Figure 4is a schematic diagram of a multi-head attention module provided in an embodiment of the present application;

[0051] Figure 5 is a schematic diagram of a scaled dot product attention module provided in an embodiment of the present application;

[0052] Figure 6 is a schematic diagram of a bypass network provided in an embodiment of the present application;

[0053] Figure 7 is a flow chart of a target detection method provided in an embodiment of the present application;

[0054] Figure 8 It is a structural schematic diagram of a model training device provided in an embodiment of the present application;

[0055] Fig. 9 is a structural schematic diagram of a target detection device provided in an embodiment of the present application;

[0056] Fig.10 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0057] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0058] It should be understood that the "multiple" mentioned in this application refers to two or more. In the description of this application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in order to facilitate the clear description of the technical solution of this application, the words "first" and "second" are used to distinguish between the same items or similar items with basically the same functions and effects. Those skilled in the art can understand that the words "first" and "second" do not limit the quantity and execution order, and the words "first" and "second" do not limit them to be different.

[0059] The phrases such as "one embodiment" or "some embodiments" described in the present application mean that the specific features, structures or characteristics described in the embodiment are included in one or more embodiments of the present application. Therefore, the phrases such as "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments" that appear in the differences in the present application do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. In addition, the terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0060] The application scenarios of the embodiments of the present application are described below.

[0061] Object detection is widely used in urban management scenarios such as intelligent transportation and intelligent monitoring. In the process of continuous urban development, many object detection needs often arise, such as pedestrian detection, vehicle detection, and manhole cover missing detection. Every time a new demand is proposed, the model training method provided in the embodiment of the present application can be used to train an object detection model to detect new objects.

[0062] The model training method provided in the embodiment of the present application can fine-tune the pre-trained large model to obtain an object detection model that can meet new requirements. In this way, only a small number of training samples are needed to complete the model training, and the overall model training takes a short time, thereby greatly reducing the model training cost.

[0063] The model training method provided in the embodiment of the present application is explained in detail below.

[0064] Figure 1 is a flow chart of a model training method provided in an embodiment of the present application. The method is applied to a computer device, which may be a device with model training capabilities, for example, the computer device may be a terminal, a server, etc., which is not limited in the embodiment of the present application. Figure 1 , the method comprises the following steps:

[0065] Step 101: A computer device obtains a pre-trained model.

[0066] The pre-training module is a pre-trained large model. The pre-training model is used to identify multiple targets. Optionally, the pre-training model can be a GLIP (Grounded Language-Image Pre-training) model. Of course, the pre-training model can also be other models that can identify multiple targets, and the embodiments of the present application are not limited to this.

[0067] For example, see Figure 2 The pre-trained model may include: a language backbone network, a vision backbone network, a deep fusion network, a word-region alignment score network, a loss network, etc.

[0068] Those skilled in the art will understand that Figure 2 This is merely an example of the pre-trained model and does not constitute a limitation of the pre-trained model. The pre-trained model may include more or fewer networks than shown in the figure.

[0069] The input of the pre-trained model is text and an image. Optionally, the text can be a prompt for the image.

[0070] The language backbone network is used to extract the language features of the input text. Optionally, the language backbone network can be a Clip (Contrastive Language-Image Pre-Training) model. Of course, the language backbone network can also be other backbone networks that can extract the language features of the text, and the embodiments of the present application are not limited to this.

[0071] The image backbone network is used to extract image features of the input image. Optionally, the image backbone network can be a Swin Transformer model, such as a Swin-Tiny model. Of course, the image backbone network can also be other backbone networks that can extract image features of an image, and the embodiments of the present application are not limited to this.

[0072] The language features extracted by the language backbone network and the image features extracted by the image backbone network are output to the deep fusion network. The deep fusion network is used to fuse the input language features and image features to obtain language features fused with image semantics and image features fused with text semantics and output them to the word region alignment score network.

[0073] The word region alignment score network is used to determine the similarity matrix of the input language features and image features, and the similarity matrix contains the similarity between each element of all elements in the language features and each element of all elements in the image features. The word region alignment score network outputs the similarity matrix to the loss network.

[0074] The loss network is used to determine the loss value based on the input similarity matrix and sample labels.

[0075] It should be noted that the language features described in the embodiments of the present application may be a language feature matrix, and the image features described may be an image feature matrix.

[0076] Next, we will explain the deep fusion network in detail:

[0077] For example, Figure 2 As shown, the deep fusion network may include: a fusion module, a text encoder, and an image encoder.

[0078] The fusion module is used to process the input feature (i.e., one of the language feature and the image feature) through the attention mechanism, and fuse the processed feature with another feature (i.e., the other of the language feature and the image feature). Optionally, the fusion module can process the input feature through a self-attention mechanism or a multi-head attention mechanism, which is not limited in the embodiments of the present application.

[0079] The text encoder is used to encode the input language features. Optionally, the text encoder can be a BERT (Bidirectional Encoder Representation from Transformers) model. Of course, the text encoder can also be other encoders that can encode language features, which is limited in the embodiments of the present application.

[0080] The image encoder is used to encode the input image features. Optionally, the image encoder may be a DyHead module. Of course, the text encoder may also be other encoders capable of encoding image features, which is limited in the embodiments of the present application.

[0081] In some embodiments, Figure 2 As shown, the fusion module may include a first fusion module and a second fusion module.

[0082] The input of the first fusion module is the language features output by the language backbone network and the image features output by the image backbone network. The first fusion module processes the input language features through the attention mechanism, adds the processed language features to the input image features, obtains the image features fused with text semantics and outputs them to the image encoder.

[0083] The input of the second fusion module is the image features output by the image backbone network and the language features output by the language backbone network. The second fusion module processes the input image features through the attention mechanism, adds the processed image features to the input language features, obtains the language features fused with image semantics and outputs them to the text encoder.

[0084] In some embodiments, Figure 2 As shown, the deep fusion network may include multiple text encoders and multiple image encoders. The fusion module may include a third fusion module and a fourth fusion module.

[0085] The input of the third fusion module is the language features output by the previous text encoder and the image features output by the previous image encoder. The third fusion module processes the input language features through the attention mechanism, adds the processed language features to the input image features, obtains the image features fused with text semantics and outputs them to the next image encoder.

[0086] The input of the fourth fusion module is the image features output by the previous image encoder and the language features output by the previous text encoder. The fourth fusion module processes the input image features through the attention mechanism, adds the processed image features to the input language features, obtains the language features fused with image semantics and outputs them to the next text encoder.

[0087] It should be noted that, in some cases, the fusion module may include only the first fusion module and the second fusion module, or only the third fusion module and the fourth fusion module. In other cases, the fusion module may include both the first fusion module and the second fusion module, and also the third fusion module and the fourth fusion module.

[0088] Another point to note is that the text features output by the last text editor in the deep fusion network are the input of the word region alignment score network. The image features output by the last image editor in the deep fusion network are the input of the word region alignment score network.

[0089] The deep fusion network in the embodiment of the present application calculates cross-attention for the language features of the text modality and the image features of the image modality, so that cross-modal feature learning can begin at a relatively shallow model stage.

[0090] Next, the fusion module is explained in detail:

[0091] For example, Figure 3 As shown, any fusion module (i.e., the first fusion module, the second fusion module, the third fusion module, or the fourth fusion module mentioned above) may include: a query layer, a key layer, a value layer, an input linear layer corresponding to the query layer, an input linear layer corresponding to the key layer, an input linear layer corresponding to the value layer, an attention module, and an output linear layer.

[0092] The inputs of the query layer, key layer, and value layer are all input features of this fusion module.

[0093] In this case, the input features are multiplied by the weight matrix of the query layer to obtain the query matrix and output it to the corresponding input linear layer, which performs a linear transformation on the input query matrix and outputs it to the attention module.

[0094] The input features are multiplied by the weight matrix of the key layer to obtain the key matrix and output it to the corresponding input linear layer. The input linear layer performs a linear transformation on the input key matrix and outputs it to the attention module.

[0095] The input features are multiplied by the weight matrix of the value layer to obtain the value matrix and output it to the corresponding input linear layer. The input linear layer performs a linear transformation on the input value matrix and outputs it to the attention module.

[0096] The attention module processes the input query matrix, key matrix, and value matrix through the attention mechanism to obtain the attention matrix and output it to the output linear layer. Optionally, the attention module can be a scaled dot-product attention module. Of course, the attention module can also be other attention modules, which is not limited in the embodiments of the present application.

[0097] The output linear layer performs a linear transformation on the input attention matrix and then outputs it.

[0098] For example, Figure 4 As shown, this fusion module can be a multi-head attention module.

[0099] There are h attention modules in the multi-head attention module, where h is an integer greater than or equal to 2.

[0100] There are h input linear layers corresponding to the query layer in the multi-head attention module. The i-th input linear layer in the h input linear layers performs a linear transformation on the input query matrix and outputs it to the i-th attention module in the h attention modules. Where i is an integer greater than or equal to 1 and less than or equal to h.

[0101] There are h input linear layers corresponding to the key layer in the multi-head attention module. The i-th input linear layer among the h input linear layers performs a linear transformation on the input key matrix and outputs it to the i-th attention module among the h attention modules.

[0102] There are h input linear layers corresponding to the value layer in the multi-head attention module. The i-th input linear layer among the h input linear layers performs a linear transformation on the input value matrix and outputs it to the i-th attention module among the h attention modules.

[0103] The multi-head attention module also includes a concatenation layer, which is located between each attention module and the output linear layer in the h attention modules.

[0104] The attention matrix output by each of the h attention modules is the input of the concatenation layer. The concatenation layer concatenates the h input attention matrices to obtain a combined attention matrix and outputs it to the output linear layer.

[0105] The output linear layer performs a linear transformation on the input merged attention matrix and outputs it.

[0106] For example, Figure 5 As shown, the scaled dot product attention module may include: a first matrix multiplication (MatMul) layer, a scaling (Scale) layer, a shielding (Mask) layer, a normalization (SoftMax) layer, and a second matrix multiplication layer.

[0107] The input of the first matrix multiplication layer is the query matrix output by the input linear layer corresponding to the query layer and the key matrix output by the input linear layer corresponding to the key layer. The first matrix multiplication layer performs matrix dot product operation on the input query matrix and key matrix to obtain the score matrix and output it to the scaling layer.

[0108] The scaling layer scales the elements in the input score matrix and outputs them to the shielding layer.

[0109] The shielding layer specifically shields the elements in the input score matrix and outputs it to the normalization layer.

[0110] The normalization layer normalizes the elements in the input score matrix to obtain the weight coefficient matrix and outputs it to the second matrix multiplication layer.

[0111] The input of the second matrix multiplication layer is the weight coefficient matrix output by the normalization layer and the value matrix output by the input linear layer corresponding to the value layer. The second matrix multiplication layer multiplies the input weight coefficient matrix with the input value matrix to obtain the attention matrix and output it.

[0112] Step 102: The computer device adds a specified bypass network to the pre-trained model to obtain a specified model.

[0113] The specified bypass network may include a bypass network of a specified layer in the fusion module, and the specified layer may include at least one of a query layer, a key layer, and a value layer in the fusion module. Optionally, the specified bypass network may also include a bypass network of an input linear layer corresponding to the specified layer in the fusion module, and / or the specified bypass network may also include a bypass network of an output linear layer in the fusion module.

[0114] For example, the specified bypass network may include a bypass network of a query layer in a fusion module, a bypass network of a value layer in a fusion module, and a bypass network of an output linear layer in a fusion module.

[0115] Alternatively, the specified bypass network may include a bypass network of a query layer in a fusion module, a bypass network of a value layer in a fusion module, a bypass network of an input linear layer corresponding to the query layer in a fusion module, and a bypass network of an input linear layer corresponding to the value layer in a fusion module.

[0116] Alternatively, the specified bypass network may include a bypass network of a query layer in a fusion module, a bypass network of a value layer in a fusion module, a bypass network of an input linear layer corresponding to the query layer in a fusion module, a bypass network of an input linear layer corresponding to the value layer in a fusion module, and a bypass network of an output linear layer in a fusion module.

[0117] The input of the bypass network of a layer (including the specified layer, input linear layer, or output linear layer in the above fusion module) is the same as the input of this layer. The output of this layer is added to the output of the bypass network of this layer as the input of the next layer of this layer. For example, the bypass network can be a LoRA (Low-Rank Adaptation) network.

[0118] For example, Figure 6 As shown, the bypass network of a certain layer includes a dimensionality reduction layer and a dimensionality increase layer. The bypass network first reduces the dimensionality of the input features and then increases the dimensionality before outputting them. Optionally, the weight matrix of the dimensionality reduction layer is initialized to obey a Gaussian distribution, and the weight matrix of the dimensionality increase layer is initialized to be a zero matrix.

[0119] Assume that the input feature is an m×d feature matrix. The weight matrix of this layer is a d×k matrix. The weight matrix of the dimension reduction layer in the bypass network can be a d×r matrix. The weight matrix of the dimension increase layer in the bypass network can be an r×k matrix, where m, d, r, and k are all positive integers, and r is less than d, and r is less than k.

[0120] In this case, the input features are multiplied by the weight matrix of this layer to obtain an m×k matrix. The input features are multiplied by the weight matrix of the dimension reduction layer in the bypass network to obtain an m×r matrix and output to the dimension increase layer in the bypass network. The m×r matrix is ​​multiplied by the weight matrix of the dimension increase layer in the bypass network to obtain an m×k matrix. The m×k matrix output by this layer is added to the m×k matrix output by the dimension increase layer in the bypass network to obtain an m×k matrix as the output feature and output to the next layer.

[0121] In some embodiments, the operation of step 102 may be: the computer device determines all designated layers in the pre-trained model according to a preset layer identifier, and constructs a bypass network for each designated layer in the pre-trained model to obtain a designated model.

[0122] The preset layer identifier can be set in advance. The preset layer identifier is used to identify the layer in the pre-trained model that needs to build a bypass network.

[0123] The preset layer identifier may include an identifier of a specified layer. For example, the preset layer identifier may include the name of the query layer in the fusion module "out_v_proj", "out_l_proj", and / or may include the name of the value layer in the fusion module "values_v_proj", "values_l_proj".

[0124] In this case, the computer device can locate all the specified layers in the pre-trained model according to the identifier of the specified layer, and then construct a bypass network for each specified layer.

[0125] Optionally, the preset layer identifier may also include an identifier of an input linear layer corresponding to the specified layer in the fusion module, and / or, may also include an identifier of an output linear layer in the fusion module. For example, the preset layer identifier may also include the name "v_proj", "l_proj" of the output linear layer in the fusion module.

[0126] When the preset layer identifier also includes the identifier of the input linear layer corresponding to the specified layer in the fusion module, the computer device can locate all the input linear layers corresponding to the specified layer in the fusion module in the pre-trained model according to the identifier of the input linear layer corresponding to the specified layer in the fusion module, and then construct a bypass network for each of the input linear layers.

[0127] When the preset layer identifier also includes the identifier of the output linear layer in the fusion module, the computer device can locate all the output linear layers in the fusion module in the pre-trained model according to the identifier of the output linear layer in the fusion module, and then construct a bypass network for each of the output linear layers.

[0128] After the computer device constructs each required designated bypass network in the pre-training module according to the preset layer identifier, the designated model is obtained.

[0129] Step 103: The computer device obtains a training sample set.

[0130] The training sample set includes multiple training samples, and each training sample includes an image sample, a text sample and a sample label.

[0131] The image sample includes a preset target. The text sample is a text description of the preset target included in the image sample. Optionally, the sample tag can be a semantic similarity matrix between the image sample and the text sample, and the semantic similarity matrix can include the semantic similarity between each region in the image sample and each word in the text sample.

[0132] The preset target can be preset. For example, the preset target can be a pedestrian, a vehicle, a manhole cover, etc., which is not limited in the embodiment of the present application.

[0133] It should be noted that, compared with the method in the related art that requires a large number of training samples to perform model training, only a small number of training samples are required in the embodiment of the present application to complete the subsequent model training. That is, the number of training samples in the training sample set in the embodiment of the present application can be relatively small, such as 500, 600, etc., and the embodiment of the present application does not limit this.

[0134] In some embodiments, the operation of step 103 may be: the computer device acquires a plurality of images, and generates the training sample set according to the plurality of images.

[0135] Each of the multiple images contains a preset target. The number of the multiple images is relatively small, such as 500, 600, etc., which is not limited in the embodiment of the present application.

[0136] In some cases, the installed equipment may be provided with a model for identifying a specific target obtained by fine-tuning a pre-trained large model. Since the pre-trained large model has learned a large amount of prior knowledge, the model obtained by fine-tuning the large model has a certain recognition ability for new targets. In this case, multiple images containing preset targets can be collected through the installed equipment.

[0137] Assume that the installed and used device is the first device. The first device may be a terminal, a server, or other devices. For example, the first device may be a monitoring device, a mobile phone, a computer, or the like, which is not limited in the present embodiment of the application.

[0138] A first model is provided in the first device. The first model is obtained by training the pre-trained model or the specified model. For example, the first model may be a model for identifying the first target previously trained using the model training method provided in the embodiment of the present application. In this case, the first model is mainly used to identify the first target, and the first model also has a certain recognition capability for other targets, that is, the first model also has a certain recognition capability for the preset target.

[0139] In this case, the first device can identify images containing the preset target through the first model during operation to obtain multiple images containing the preset target. Optionally, the computer device can send an image search request for a category containing the preset target to the first device. After receiving the image search request, the first device can identify multiple images containing the preset target from all images acquired by itself, and then send the multiple images to the computer device.

[0140] For example, the first device may be a monitoring device. The first device may shoot a monitoring video. The first device may perform target detection on the video image of the monitoring video through the first model to determine whether the video image contains a preset target, and then the first device may send multiple video images containing the preset target to the computer device.

[0141] In the embodiment of the present application, the first device can be used to automatically screen out multiple images containing preset targets, and then handed over to technicians for corresponding annotation. Compared with the method in the related art that requires technicians to manually screen images, the embodiment of the present application can greatly reduce labor costs.

[0142] Optionally, the operation of the computer device generating the training sample set according to the multiple images may include the following two methods.

[0143] The first method is: for any one of the multiple images, first annotation information of the image is obtained, and the first annotation information includes the target area position and the target category; the image is used as an image sample of a training sample, the target category is used as a text sample of the training sample, and the sample label of the training sample is determined according to the first annotation information to obtain the training sample.

[0144] The first annotation information may be the annotation information of the image by the technician. The technician may manually select the target area in the image and mark the corresponding target category. The target area is the area where the preset target exists, and the target area may generally be a rectangular area. The target category is the category of the preset target, and the target category is presented in text form.

[0145] Optionally, the first annotation information may indicate that the semantic similarity between the target region in the image and the target category is 1, and the semantic similarity between other regions in the image except the target region and the target category is 0. In this case, the computer device may determine a semantic similarity matrix between the image and the target category as a sample label of the training sample based on the first annotation information, and the semantic similarity matrix includes the semantic similarity between each region in the image and each word in the target category.

[0146] The second method: for any one of the multiple images, obtain first annotation information and text description information of the image, the first annotation information includes the target area position and the target category, and the text description information is a description of the image; obtain second annotation information of the text description information, the second annotation information includes the target word position and the target category; use the image as an image sample of a training sample, use the text description information as a text sample of the training sample, and determine the sample label of the training sample according to the first annotation information and the second annotation information to obtain the training sample.

[0147] The first annotation information may be the annotation information of the image by the technician. The technician may manually select the target area in the image and mark the corresponding target category. The target area is the area where the preset target exists, and the target area may generally be a rectangular area. The target category is the category of the preset target.

[0148] The text description information may be a description of the image by a technician, and may include a description of the category, appearance, etc. of a preset target contained in the image.

[0149] The second annotation information may be annotation information of the text description information by the technician. The technician may manually select the target word in the text description information and mark the corresponding target category. The target word is a description of the preset target. The target category is the category of the preset target.

[0150] Optionally, the first annotation information and the second annotation information may indicate that: the semantic similarity between the target area in the image and the target word in the text description information is 1, the semantic similarity between the target area in the image and other words in the text description information except the target word is 0, the semantic similarity between other areas in the image except the target area and the target word in the text description information is 0, and the semantic similarity between other areas in the image except the target area and other words in the text description information except the target word is 0. In this case, the computer device may determine a semantic similarity matrix between the image and the text description information as a sample label of the training sample based on the first annotation information and the second annotation information, and the semantic similarity matrix includes the semantic similarity between each area in the image and each word in the text description information.

[0151] Step 104: The computer device uses the training sample set to train the specified model to obtain a target detection model; wherein, during the training process, the weights in the specified model except for the weights of the specified bypass network are frozen.

[0152] The target detection model is a model used to identify preset targets.

[0153] In the embodiment of the present application, the specified model is obtained by adding a specified bypass network to the pre-trained model. In the process of training the specified model, all the original weights in the pre-trained model are frozen, that is, all the original weights in the pre-trained model remain unchanged, and only the weights of the specified bypass network are adjusted.

[0154] The fusion module in the pre-trained model is used to fuse language features and image features, which can realize the semantic association between language features and image features. The specified bypass network added in the pre-trained model includes a bypass network of at least one of the query layer, key layer, and value layer in the fusion module, and further includes a bypass network of the input linear layer and / or output linear layer in the fusion module. Therefore, when the pre-trained model already has a certain recognition ability for the preset target, the adjustment of the weight of the specified bypass network during the training process can strengthen the semantic association between language features and image features, and can also speed up the establishment of a matching relationship between text and image. In this way, only the weight of the specified bypass network is adjusted during the training process, and a target detection model that can accurately identify the preset target can be obtained.

[0155] Since only the weights of the specified bypass network need to be adjusted during the training process, not only can the weights that need to be adjusted during the model training process be greatly reduced, but only a small number of training samples are needed to complete the model training. In this way, the time and computing power required for model training can be greatly reduced, thereby reducing the cost of model training.

[0156] In some embodiments, when the computer device uses the training sample set to train the specified model, for each training sample in the training sample set, the text sample and the image sample in the training sample can be input into the specified model to obtain output data (such as the similarity matrix output by the above-mentioned word region alignment score network); the loss value between the output data and the sample label in the training sample is determined by the loss function; and the weight of the specified bypass network in the specified model is adjusted according to the loss value. After the weight of the specified bypass network in the specified model is adjusted based on each training sample in the training sample set, the adjusted specified model is the target detection model.

[0157] Among them, the operation of the computer device adjusting the weight of the specified bypass network in the specified model according to the loss value is similar to the operation of adjusting the weight of a layer in the neural network model according to the loss value in the related art, and the embodiment of the present application will not elaborate on this.

[0158] For example, the computer device can be To adjust any weight in the weight matrix of the specified bypass network in the specified model. is the weight after adjustment. w is the weight before adjustment. α is the learning rate, which can be preset, such as α can be 0.001, 0.000001, etc., and the present application embodiment does not make a sole limitation on this. dw is the partial derivative of the loss function with respect to w, which can be obtained according to the loss value.

[0159] In some embodiments, the computer device may also obtain a test sample set, the test sample set including a plurality of test samples, each of the plurality of test samples including a text sample, an image sample and a sample tag. The image sample includes a preset target. The text sample is a text description of the preset target included in the image sample. By way of example, the sample tag may be a semantic similarity matrix between the image sample and the text sample.

[0160] Optionally, the number of test samples in the test sample set may be the same as the number of training samples in the training sample set. Of course, the number of test samples in the test sample set may be different from the number of training samples in the training sample set, but the difference between the two may be small.

[0161] The way in which the computer device obtains the test sample set is similar to the way in which the computer device obtains the training sample set in the above step 103, and this embodiment of the present application will not be repeated here.

[0162] The computer device can use the training sample set to evaluate the generalization ability of the target detection model (including accuracy, recall rate, etc.). After the target detection model is evaluated, the target detection model can be deployed to related devices so that the related devices can perform target detection through the target detection model.

[0163] Assume that the second device has a detection requirement for a preset target. The second device may be a terminal, a server, or other device. For example, the second device may be a monitoring device, a mobile phone, a computer, or the like, which is not limited in the present embodiment of the application.

[0164] In a possible manner, if the model has not been deployed in the second device, the computer device can send the target detection model to the second device. After the second device receives the target detection model, it can detect the preset target through the target detection model.

[0165] In another possible manner, a second model is deployed in the second device, and the second model is a model for identifying a second target obtained by training using the model training method provided in an embodiment of the present application. The second target may be the same as or different from the preset target. Then the computer device can generate a weight file carrying the weights of the specified bypass network in the target detection model, and send the weight file to the second device. After the second device receives the weight file, it can update the second model set in the second device according to the weight file. Specifically, the weights of the specified bypass network in the second model can be updated to the weights carried in the weight file. In this way, the updated second model is the target detection model, which can be used to detect the preset target.

[0166] Since the weight file only carries the weights of the specified bypass network in the target detection model, the file size of the weight file is relatively small, such as only 8MB (megabytes). Since the file size of the weight file is relatively small, the computer device sends the weight file to the second device at a relatively fast speed, and the second device updates the second model according to the weight file at a relatively fast speed, thereby greatly improving the model deployment speed.

[0167] In an embodiment of the present application, a pre-trained model is obtained, and then a specified bypass network is added to the pre-trained model to obtain a specified model. Afterwards, the specified model is trained using a training sample set to obtain a target detection model. Among them, all the original weights in the pre-trained model are frozen during the training process, and only the weights of the specified bypass network are adjusted. Since the fusion module in the pre-trained model is used to fuse language features and image features, and the specified bypass network includes at least one bypass network in the query layer, key layer, and value layer in the fusion module, the adjustment of the weight of the specified bypass network during the training process can strengthen the semantic association between language features and image features, and a target detection model with a higher accuracy can be obtained. In addition, since only the weight of the specified bypass network needs to be adjusted during the training process, not only can the weights that need to be adjusted during the model training process be greatly reduced, but also only a small number of training samples are required to complete the model training. In this way, the time and computing power required for model training can be greatly reduced, thereby reducing the cost of model training.

[0168] It should be noted that the model training method provided in the embodiments of the present application can be applicable to a variety of scenarios.

[0169] In one possible scenario, a device previously had no detection requirements and no model was deployed. However, later on, this device has a detection requirement for a preset target. In this case, a target detection model can be trained using the model training method provided in the embodiment of the present application and deployed in this device, and this device can detect the preset target through the target detection model.

[0170] In another possible scenario, a certain device has previously deployed a model, which is used to detect targets different from preset targets, and the model is previously trained by the model training method provided in the embodiment of the present application. Later, this device wants to switch to detecting preset targets. In this case, the target detection model can be trained by the model training method provided in the embodiment of the present application, and then a weight file carrying the weights of the specified bypass network in the target detection model is sent to this device. This device can update the weights of the specified bypass network in the model set by itself to the weights carried in the weight file, and then the detection of the preset target can be achieved through the model.

[0171] As can be seen from the above, the embodiment of the present application can realize the switching of detection targets by loading different weight files in the device where the model has been deployed. For example, if a device that has deployed a pedestrian detection model and is used for pedestrian detection is to be switched to a device for vehicle detection, it is only necessary to load the weight file of the specified bypass network in the vehicle detection model in the device to update the weight of the specified bypass network in the pedestrian detection model to the weight of the specified bypass network in the vehicle detection model. In this way, the pedestrian detection model can be updated to the vehicle detection model, thereby meeting the latest detection requirements of the device.

[0172] In another possible scenario, a certain device has previously deployed a model, which is used to detect a preset target, and the model is previously trained by the model training method provided by the embodiment of the present application. The installation position of the device has changed later. For example, the device was originally installed at a position three meters away from the ground, and then installed at a position five meters away from the ground. Then the appearance characteristics of the preset target captured by the device are likely to be somewhat different from before. In this case, the device can first identify multiple images containing preset targets in the images captured by itself through the model set by itself, and then send the multiple images to the computer device. The computer device can obtain a target detection model based on the multiple images through the model training method provided by the embodiment of the present application, and then send the weight file carrying the weight of the specified bypass network in the target detection model to the device. The device can update the weight of the specified bypass network in the model set by itself to the weight carried in the weight file, and the updated model is more suitable for identifying the preset target captured by the device at the current position. The device can more accurately detect the preset target through the updated model.

[0173] From the above, it can be seen that in the embodiment of the present application, loading different weight files in the device where the model has been deployed can achieve accurate detection of targets photographed at different locations.

[0174] It should be noted that the target detection model trained in the embodiment of the present application can be applied to the city management system to detect various targets. For example, the target detection model trained in the embodiment of the present application can be applied to various monitoring devices in the city management system to achieve various detection requirements such as pedestrian detection, vehicle detection, and manhole cover loss detection.

[0175] In the related art, the detection algorithms in the city management system are numerous and messy, and it is difficult to quickly adapt to new detection requirements. However, the model training method in the embodiment of the present application can be applied to various detection scenarios, and the model training cost is low, the model deployment speed is fast, and it can quickly adapt to the proposed new detection requirements.

[0176] The following is an explanation of the process of target detection using the above target detection model:

[0177] Figure 7 1 is a flow chart of a target detection method provided in an embodiment of the present application. The method can be applied to a third device, which can be a terminal, a server, etc., and the embodiment of the present application does not limit this.

[0178] See also Figure 7 , the method comprises the following steps:

[0179] Step 701: The third device obtains an image to be detected.

[0180] The image is an image for which target detection is required. For example, the image may be an image captured by a third device, for example, the image may be a video image in a video captured by the third device.

[0181] In some embodiments, after receiving the detection instruction, the third device may execute steps 701 to 702 to perform target detection.

[0182] The detection instruction may be triggered by a user in the third device, or the detection instruction may be sent to the third device by other devices.

[0183] The detection instruction is used to instruct to detect a preset target. Optionally, the detection instruction may carry the category of the preset target, or may carry text description information of the preset target.

[0184] Step 702: The third device determines whether the image contains a preset target based on the target detection model.

[0185] The target detection model is obtained through the above Figure 1 The model is obtained by training using the model training method provided in the embodiment.

[0186] The third device can input the image into the target detection model, and input the category of the preset target or the text description information of the preset target into the target detection model, and the target detection model outputs the detection result, which is used to indicate whether the image contains the preset target.

[0187] For example, the detection result may include the position of a target frame in the image. The target frame is used to indicate an area in the image where a preset target exists, and the area is usually a rectangular area.

[0188] In this case, if the image contains a preset target, a target frame will appear in the image to select the preset target; if the image does not contain the preset target, the target frame will not be displayed in the image. In this way, the user can quickly know whether there is a preset target in the image, which improves the user experience.

[0189] In an embodiment of the present application, after the third device acquires the image to be detected, it can determine whether the image contains a preset target based on the target detection model, thereby quickly and accurately realizing target detection of the image.

[0190] Figure 8 is a schematic diagram of the structure of a model training device provided in an embodiment of the present application. The device can be implemented as part or all of a computer device by software, hardware, or a combination of both. The computer device can be the following Fig.10 Computer equipment shown. Figure 8 The device includes: an acquisition module 801, an addition module 802, and a training module 803.

[0191] An acquisition module 801 is used to acquire a pre-trained model. The pre-trained model is used to identify multiple targets. The pre-trained model includes a language backbone network, an image backbone network, and a deep fusion network. The language backbone network is used to extract language features. The image backbone network is used to extract image features. The deep fusion network is used to fuse language features and image features. The deep fusion network includes a fusion module.

[0192] An adding module 802 is used to add a specified bypass network to the pre-trained model to obtain a specified model, wherein the specified bypass network includes a bypass network of a specified layer in the fusion module, and the specified layer includes at least one of a query layer, a key layer, and a value layer in the fusion module;

[0193] The training module 803 is used to train the specified model using the training sample set to obtain the target detection model; wherein, during the training process, the weights in the specified model except the weights of the specified bypass network are frozen.

[0194] Optionally, the specified bypass network also includes a bypass network of an input linear layer corresponding to the specified layer in the fusion module, and / or the specified bypass network also includes a bypass network of an output linear layer in the fusion module.

[0195] Optionally, the adding module 802 is used for:

[0196] Determine all specified layers in the pre-trained model according to the preset layer identifier;

[0197] A bypass network is constructed for each specified layer in the pre-trained model to obtain the specified model. The bypass network includes a dimensionality reduction layer and a dimensionality increase layer. The weight matrix of the dimensionality reduction layer obeys a Gaussian distribution after initialization, and the weight matrix of the dimensionality increase layer is initialized to a zero matrix.

[0198] Optionally, the device further comprises:

[0199] A receiving module, used for receiving a plurality of images sent by a first device, wherein a first model is provided in the first device, the first model is obtained by training a pre-trained model or a specified model, and each of the plurality of images is an image containing a preset target recognized by the first device through the first model;

[0200] The first generating module is used to generate a training sample set according to a plurality of images, wherein each training sample in the training sample set includes an image sample, a text sample and a sample label.

[0201] Optionally, the first generating module is used for:

[0202] For any one of the multiple images, first annotation information of the image is obtained, where the first annotation information includes a target region position and a target category;

[0203] An image is taken as an image sample of a training sample, a target category is taken as a text sample of a training sample, and a sample label of a training sample is determined according to the first annotation information to obtain a training sample.

[0204] Optionally, the first generating module is used for:

[0205] For any one of the multiple images, first annotation information and text description information of the image are obtained, where the first annotation information includes a target region position and a target category, and the text description information is a description of the image;

[0206] Acquire second annotation information of the text description information, the second annotation information including a target word position and a target category;

[0207] An image is taken as an image sample of a training sample, text description information is taken as a text sample of a training sample, and a sample label of a training sample is determined according to the first labeling information and the second labeling information to obtain a training sample.

[0208] Optionally, the device further comprises:

[0209] A second generation module is used to generate a weight file carrying the weights of a specified bypass network in the target detection model;

[0210] The sending module is used to send a weight file to the second device, where the weight file is used to update the second model set in the second device.

[0211] In an embodiment of the present application, a pre-trained model is obtained, and then a specified bypass network is added to the pre-trained model to obtain a specified model. Afterwards, the specified model is trained using a training sample set to obtain a target detection model. Among them, all the original weights in the pre-trained model are frozen during the training process, and only the weights of the specified bypass network are adjusted. Since the fusion module in the pre-trained model is used to fuse language features and image features, and the specified bypass network includes at least one bypass network in the query layer, key layer, and value layer in the fusion module, the adjustment of the weight of the specified bypass network during the training process can strengthen the semantic association between language features and image features, and a target detection model with a higher accuracy can be obtained. In addition, since only the weight of the specified bypass network needs to be adjusted during the training process, not only can the weights that need to be adjusted during the model training process be greatly reduced, but also only a small number of training samples are required to complete the model training. In this way, the time and computing power required for model training can be greatly reduced, thereby reducing the cost of model training.

[0212] It should be noted that: the model training device provided in the above embodiment only uses the division of the above-mentioned functional modules as an example during model training. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0213] The functional units and modules in the above embodiments may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit, and the above integrated units may be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present application.

[0214] The model training device and the model training method embodiments provided in the above embodiments belong to the same concept. The specific working process of the units and modules in the above embodiments and the technical effects brought about can be found in the method embodiment part and will not be repeated here.

[0215] Fig. 9 is a schematic diagram of the structure of a target detection device provided in an embodiment of the present application. The device can be implemented by software, hardware or a combination of both to form part or all of a computer device, and the computer device can be as follows Fig.10 Computer equipment shown. Fig. 9 The device includes: an acquisition module 901 and a detection module 902.

[0216] An acquisition module 901 is used to acquire an image to be detected;

[0217] The detection module 902 is used to determine whether the image contains a preset target based on a target detection model, and the target detection model is trained by the above-mentioned model training method.

[0218] In an embodiment of the present application, after acquiring an image to be detected, it is possible to determine whether the image contains a preset target based on a target detection model, thereby enabling rapid and accurate target detection of the image.

[0219] It should be noted that: the target detection device provided in the above embodiment only uses the division of the above-mentioned functional modules as an example for target detection. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0220] The functional units and modules in the above embodiments may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit, and the above integrated units may be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present application.

[0221] The target detection device and target detection method embodiments provided in the above embodiments belong to the same concept. The specific working process of the units and modules in the above embodiments and the technical effects brought about can be found in the method embodiment part and will not be repeated here.

[0222] Fig.10 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. Fig.10 As shown, the computer device 10 includes: a processor 100, a memory 101, and a computer program 102 stored in the memory 101 and executable on the processor 100. When the processor 100 executes the computer program 102, the steps in the model training method or target detection method in the above-mentioned embodiments are implemented.

[0223] The computer device 10 may be a general-purpose computer device or a special-purpose computer device. In a specific implementation, the computer device 10 may be a desktop computer, a portable computer, a network server, a PDA, a mobile phone, a tablet computer, a wireless terminal device, a communication device or an embedded device. The embodiment of the present application does not limit the type of the computer device 10. Those skilled in the art will understand that Fig.10 This is only an example of the computer device 10 and does not constitute a limitation on the computer device 10. The computer device 10 may include more or fewer components than shown in the figure, or a combination of certain components, or different components, such as input and output devices, network access devices, etc.

[0224] The processor 100 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0225] In some embodiments, the memory 101 may be an internal storage unit of the computer device 10, such as a hard disk or memory of the computer device 10. In other embodiments, the memory 101 may also be an external storage device of the computer device 10, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the computer device 10. Further, the memory 101 may also include both an internal storage unit of the computer device 10 and an external storage device. The memory 101 is used to store an operating system, an application program, a boot loader, data, and other programs. The memory 101 may also be used to temporarily store data that has been output or is to be output.

[0226] An embodiment of the present application also provides a computer device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, and when the processor executes the computer program, the steps in any of the above-mentioned method embodiments are implemented.

[0227] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0228] An embodiment of the present application provides a computer program product, which, when executed on a computer, enables the computer to execute the steps in the above-mentioned method embodiments.

[0229] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above method embodiments, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above method embodiments can be implemented. Among them, the computer program includes computer program code, which can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the camera / terminal device, recording medium, computer memory, ROM (Read-Only Memory), RAM (Random Access Memory), CD-ROM (Compact Disc Read-Only Memory), magnetic tape, floppy disk and optical data storage device. The computer-readable storage medium mentioned in the present application can be a non-volatile storage medium, in other words, it can be a non-transient storage medium.

[0230] It should be understood that all or part of the steps to implement the above embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. The computer instructions can be stored in the above-mentioned computer readable storage medium.

[0231] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0232] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0233] In the embodiments provided in the present application, it should be understood that the disclosed devices / computer equipment and methods can be implemented in other ways. For example, the device / computer equipment embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0234] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0235] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A model training method, It is characterized in that The method comprises: Obtain a pre-trained model, the pre-trained model is used to identify multiple targets, the pre-trained model includes a language backbone network, an image backbone network, and a deep fusion network, the language backbone network is used to extract language features, the image backbone network is used to extract image features, the deep fusion network is used to fuse the language features and the image features, and the deep fusion network includes a fusion module; Adding a specified bypass network to the pre-trained model to obtain a specified model, wherein the specified bypass network includes a bypass network of a specified layer in the fusion module, and the specified layer includes at least one of a query layer, a key layer, and a value layer in the fusion module; The designated model is trained using a training sample set to obtain a target detection model; wherein, during the training process, weights in the designated model except for weights of the designated bypass network are frozen.

2. The method according to claim 1, It is characterized in that The designated bypass network also includes a bypass network of an input linear layer in the fusion module corresponding to the designated layer, and / or the designated bypass network also includes a bypass network of an output linear layer in the fusion module.

3. The method according to claim 1, It is characterized in that The step of adding a specified bypass network to the pre-trained model to obtain a specified model includes: Determine all the designated layers in the pre-trained model according to the preset layer identifier; A bypass network is constructed for each of the designated layers in the pre-trained model to obtain the designated model, wherein the bypass network includes a dimensionality reduction layer and a dimensionality increase layer, wherein the weight matrix of the dimensionality reduction layer obeys a Gaussian distribution after being initialized, and the weight matrix of the dimensionality increase layer is a zero matrix after being initialized.

4. The method according to claim 1, It is characterized in that Before using the training sample set to train the specified model, the method further includes: Receiving a plurality of images sent by a first device, wherein a first model is provided in the first device, the first model is obtained by training the pre-trained model or the specified model, and each of the plurality of images is an image containing a preset target recognized by the first device through the first model; The training sample set is generated according to the multiple images, and each training sample in the training sample set includes an image sample, a text sample and a sample label.

5. The method according to claim 4, It is characterized in that The step of generating the training sample set according to the plurality of images comprises: For any one of the multiple images, obtaining first annotation information of the image, where the first annotation information includes a target area position and a target category; The one image is used as an image sample of a training sample, the target category is used as a text sample of the one training sample, and a sample label of the one training sample is determined according to the first annotation information to obtain the one training sample.

6. The method according to claim 4, It is characterized in that The step of generating the training sample set according to the plurality of images comprises: For any one of the multiple images, obtaining first annotation information and text description information of the image, wherein the first annotation information includes a target region position and a target category, and the text description information is a description of the image; Acquire second annotation information of the text description information, wherein the second annotation information includes a target word position and a target category; The one image is used as an image sample of a training sample, the text description information is used as a text sample of the one training sample, and a sample label of the one training sample is determined according to the first annotation information and the second annotation information to obtain the one training sample.

7. The method according to any one of claims 1 to 6, It is characterized in that After the specified model is trained using the training sample set to obtain the target detection model, the method further includes: Generate a weight file carrying the weights of the specified bypass network in the target detection model; The weight file is sent to the second device, where the weight file is used to update the second model set in the second device.

8. A target detection method, It is characterized in that The method comprises: Acquire an image to be detected; Determine whether the image contains a preset target according to a target detection model, wherein the target detection model is trained by the method described in any one of claims 1 to 7.

9. A computer device, It is characterized in that The computer device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the method according to any one of claims 1 to 8 when executed by the processor.

10. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images