A model updating method and related device

Abstract

The present application relates to the field of artificial intelligence, and discloses a model updating method, including: performing multiple first clustering processes on multiple training images according to multiple target features to obtain multiple first clustering results, each of the multiple first clustering results corresponds to a silhouette coefficient, and the silhouette coefficient represents the clustering quality; determining the first target clustering result according to the silhouette coefficient, and the first target clustering result includes M clustering categories; performing second clustering processes on multiple training images according to multiple target features to obtain second clustering results, and the number of clustering categories included in the second clustering result is M. The present application does not need to verify the model corresponding to the obtained number of different clustering categories based on labeled samples to determine the optimal number of clusters, and can directly select the optimal number of clustering categories through the results of the first clustering.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a model updating method and related devices. Background Art

[0002] With the increasing attention paid to public safety and the development of video acquisition technology and large-scale data storage technology, a large number of surveillance cameras are used in crowded places where public safety incidents are likely to occur. It is difficult for humans to cope with the massive growth of surveillance videos. Therefore, the need to use computers to re-identify pedestrians in surveillance videos has emerged. Re-identification refers to identifying a specific pedestrian in the surveillance video that has appeared in the surveillance network. For example, in the surveillance network, pedestrians will appear under different cameras. The task of pedestrian re-identification is to identify the identity of pedestrians in the surveillance network and to identify the target pedestrians who have appeared in the surveillance network again in the scenes of other cameras, that is, how to re-identify pedestrians given some candidate targets.

[0003] Thanks to the development of deep learning, the performance of image retrieval and recognition tasks has been greatly improved. However, existing deep learning solutions rely on data. In the same scenario, a large amount of training data can be used to obtain a deep model with excellent performance in the current scenario. However, when the deep model is actually deployed, the distribution of the scene and the training data is inconsistent, and the performance of the deployment scenario will drop significantly. In order to solve the problem of performance degradation when deploying the target scene, unlabeled sample pairs of the target scene can be used for training to improve the performance of the target scene.

[0004] In the prior art, in the process of training using unlabeled sample pairs of the target scene, each sample is initially regarded as a separate cluster category, and each iteration reduces a certain number of cluster categories according to the minimum distance principle. As the iteration proceeds, the number of cluster categories decreases, and the model gradually improves with the training performance. Finally, the models corresponding to the different numbers of cluster categories are verified to determine the model with the best performance. In the above verification process, it is necessary to verify the models corresponding to the different numbers of cluster categories based on labeled samples. However, in some scenarios, labeled samples are often difficult to obtain. Summary of the invention

[0005] The present application provides a model updating method, the method comprising:

[0006] Get a neural network model and multiple training images;

[0007] Performing feature extraction on the plurality of training images by using the neural network model to obtain a plurality of target features;

[0008] Performing a plurality of first clustering processes on the plurality of training images according to the plurality of target features to obtain a plurality of first clustering results, wherein each of the plurality of first clustering results corresponds to a silhouette coefficient, and the silhouette coefficient represents clustering quality;

[0009] Determining a first target clustering result from the plurality of first clustering results according to a plurality of silhouette coefficients corresponding to the plurality of first clustering results, wherein the first target clustering result includes M clustering categories;

[0010] A second clustering process is performed on the multiple training images according to the multiple target features to obtain a second clustering result, and the parameters of the neural network are updated according to the first loss function, and the number of clustering categories included in the second clustering result is M.

[0011] In an optional design of the first aspect, the silhouette coefficient is calculated based on inter-class separation and intra-class cohesion, wherein the inter-class separation represents the distance between different cluster categories, and the intra-class cohesion represents the distance between different training images corresponding to the same cluster category.

[0012] In an optional design of the first aspect, the silhouette coefficient corresponding to the first target clustering result is greater than the silhouette coefficients corresponding to other clustering results among the multiple first clustering results except the first target clustering result.

[0013] In an optional design of the first aspect, performing a plurality of first clustering processes on the plurality of training images according to the plurality of target features comprises:

[0014] The plurality of training images are subjected to multiple first clustering processes according to the plurality of target features, and the parameters of the neural network are updated based on a second loss function after each first clustering process.

[0015] In an optional design of the first aspect, updating the parameters of the neural network based on the second loss function after each first clustering process includes:

[0016] After each first clustering process, the parameters of the neural network are updated based on the second loss function to reduce the variance of the first similarity distribution between each training image of the multiple training images and the multiple clustering categories included in the first clustering result in the first clustering result obtained by the next first clustering process.

[0017] In an optional design of the first aspect, the multiple first clustering processes include an i-th clustering process and a j-th clustering process, the j-th clustering process is performed after the i-th clustering process, and the performing the multiple first clustering processes on the multiple training images according to the multiple target features includes:

[0018] When performing the i-th clustering process and the j-th clustering process, obtaining a second similarity between each target feature in the multiple target features, the second similarity including an image feature similarity between each target feature in the multiple target features and a spatiotemporal constraint weight, wherein the spatiotemporal constraint weight when performing the j-th clustering process is greater than the spatiotemporal constraint weight when performing the i-th clustering process, and the spatiotemporal constraint weight represents the influence of the spatiotemporal constraint on the second similarity between the target features;

[0019] The i-th clustering process and the j-th clustering process are performed on the multiple training images according to the second similarity, wherein i and j are positive integers.

[0020] In an optional design of the first aspect, performing a plurality of first clustering processes on the plurality of training images according to the plurality of target features comprises:

[0021] Acquire a second similarity between each target feature in the plurality of target features, wherein the second similarity includes an image feature similarity and a spatiotemporal constraint weight between each target feature in the plurality of target features;

[0022] The plurality of training images are subjected to a plurality of first clustering processes according to the second similarity, and after each first clustering process, the spatiotemporal constraint weights between each target feature in the plurality of target features are updated according to the corresponding first clustering result.

[0023] In an optional design of the first aspect, the second loss function includes a ternary loss function.

[0024] In an optional design of the first aspect, updating the parameters of the neural network according to the first loss function includes:

[0025] The parameters of the neural network are updated according to the ternary loss function and the second loss function to reduce the variance of the first similarity distribution between each of the multiple training images and the multiple clustering categories included in the second clustering result obtained by the next second clustering processing, and to increase the distance between the different clustering categories in the second clustering result obtained by the next clustering processing.

[0026] In a second aspect, the present application provides a model updating device, the device comprising:

[0027] An acquisition module, used for acquiring a neural network model and a plurality of training images;

[0028] A feature extraction module, used to extract features from the plurality of training images through the neural network model to obtain a plurality of target features;

[0029] a first clustering module, configured to perform a plurality of first clustering processes on the plurality of training images according to the plurality of target features to obtain a plurality of first clustering results, wherein each of the plurality of first clustering results corresponds to a silhouette coefficient, and the silhouette coefficient represents clustering quality;

[0030] A determination module, configured to determine a first target clustering result from the plurality of first clustering results according to a plurality of silhouette coefficients corresponding to the plurality of first clustering results, wherein the first target clustering result includes M clustering categories;

[0031] The second clustering module is used to perform a second clustering process on the multiple training images according to the multiple target features to obtain a second clustering result, and update the parameters of the neural network according to the first loss function, wherein the number of clustering categories included in the second clustering result is M.

[0032] In an optional design of the second aspect, the silhouette coefficient is calculated based on inter-class separation and intra-class cohesion, wherein the inter-class separation represents the distance between different cluster categories, and the intra-class cohesion represents the distance between different training images corresponding to the same cluster category.

[0033] In an optional design of the second aspect, the silhouette coefficient corresponding to the first target clustering result is greater than the silhouette coefficients corresponding to other clustering results among the multiple first clustering results except the first target clustering result.

[0034] In an optional design of the second aspect, the first clustering module is specifically configured to:

[0035] The plurality of training images are subjected to multiple first clustering processes according to the plurality of target features, and the parameters of the neural network are updated based on a second loss function after each first clustering process.

[0036] In an optional design of the second aspect, the first clustering module is specifically configured to:

[0037] After each first clustering process, the parameters of the neural network are updated based on the second loss function to reduce the variance of the first similarity distribution between each training image of the multiple training images and the multiple clustering categories included in the first clustering result in the first clustering result obtained by the next first clustering process.

[0038] In an optional design of the second aspect, the multiple first clustering processes include an i-th clustering process and a j-th clustering process, the j-th clustering process is performed after the i-th clustering process, and the first clustering module is specifically used to:

[0039] When performing the i-th clustering process and the j-th clustering process, obtaining a second similarity between each target feature in the multiple target features, the second similarity including an image feature similarity between each target feature in the multiple target features and a spatiotemporal constraint weight, wherein the spatiotemporal constraint weight when performing the j-th clustering process is greater than the spatiotemporal constraint weight when performing the i-th clustering process, and the spatiotemporal constraint weight represents the influence of the spatiotemporal constraint on the second similarity between the target features;

[0040] The i-th clustering process and the j-th clustering process are performed on the multiple training images according to the second similarity, wherein i and j are positive integers.

[0041] In an optional design of the second aspect, the first clustering module is specifically configured to:

[0042] Acquire a second similarity between each target feature in the plurality of target features, wherein the second similarity includes an image feature similarity and a spatiotemporal constraint weight between each target feature in the plurality of target features;

[0043] The plurality of training images are subjected to a plurality of first clustering processes according to the second similarity, and after each first clustering process, the spatiotemporal constraint weights between each target feature in the plurality of target features are updated according to the corresponding first clustering result.

[0044] In an optional design of the second aspect, the second loss function includes a ternary loss function.

[0045] In an optional design of the second aspect, the second clustering module is specifically configured to:

[0046] The parameters of the neural network are updated according to the ternary loss function and the second loss function to reduce the variance of the first similarity distribution between each of the multiple training images and the multiple clustering categories included in the second clustering result obtained by the next second clustering processing, and to increase the distance between the different clustering categories in the second clustering result obtained by the next clustering processing.

[0047] In a third aspect, an embodiment of the present application provides a training device, which may include a memory, a processor, and a bus system, wherein the memory is used to store a program, and the processor is used to execute the program in the memory, including the following steps:

[0048] Get a neural network model and multiple training images;

[0049] Performing feature extraction on the plurality of training images by using the neural network model to obtain a plurality of target features;

[0050] Performing a plurality of first clustering processes on the plurality of training images according to the plurality of target features to obtain a plurality of first clustering results, wherein each of the plurality of first clustering results corresponds to a silhouette coefficient, and the silhouette coefficient represents clustering quality;

[0051] Determining a first target clustering result from the plurality of first clustering results according to a plurality of silhouette coefficients corresponding to the plurality of first clustering results, wherein the first target clustering result includes M clustering categories;

[0052] A second clustering process is performed on the multiple training images according to the multiple target features to obtain a second clustering result, and the parameters of the neural network are updated according to the first loss function, and the number of clustering categories included in the second clustering result is M.

[0053] In the third aspect of the present application, the processor can also be used to execute the steps performed by the execution device in various possible implementation methods of the first aspect. The details can be referred to the first aspect and will not be repeated here.

[0054] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer executes the model updating method described in any one of the first aspects above.

[0055] In a fifth aspect, an embodiment of the present application provides a computer program, which, when executed on a computer, enables the computer to execute any model updating method described in the first aspect.

[0056] In a sixth aspect, the present application provides a chip system, which includes a processor for supporting an execution device or a training device to implement the functions involved in the above aspects, for example, sending or processing the data and / or information involved in the above methods. In one possible design, the chip system also includes a memory, which is used to store program instructions and data necessary for the execution device or the training device. The chip system can be composed of chips, or it can include chips and other discrete devices.

[0057] The embodiment of the present application provides a model updating method, the method comprising: obtaining a neural network model and a plurality of training images; performing feature extraction on the plurality of training images through the neural network model to obtain a plurality of target features; performing a plurality of first clustering processes on the plurality of training images according to the plurality of target features to obtain a plurality of first clustering results, each of the plurality of first clustering results corresponding to a silhouette coefficient, the silhouette coefficient indicating the clustering quality; determining a first target clustering result from the plurality of first clustering results according to the plurality of silhouette coefficients corresponding to the plurality of first clustering results, the first target clustering result including M clustering categories; performing a second clustering process on the plurality of training images according to the plurality of target features to obtain a second clustering result, and updating the parameters of the neural network according to the first loss function, the number of clustering categories included in the second clustering result being the M. In the above manner, it is not necessary to determine the optimal number of clusters by verifying the models corresponding to the obtained numbers of different clustering categories based on labeled samples, and the optimal number of clustering categories can be directly selected by the result of the first clustering. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 A structural diagram of the main framework of artificial intelligence;

[0059] Figure 2 This is a schematic diagram of an application scenario architecture of this application;

[0060] Figure 3 An example of a model updating method provided in an embodiment of the present application is shown;

[0061] Figure 4 A schematic diagram of a model updating method provided in an embodiment of the present application;

[0062] Figure 5 A schematic diagram of a system architecture provided for an embodiment of the present application;

[0063] Figure 6 A schematic diagram of a structure of a model updating device provided in an embodiment of the present application;

[0064] Figure 7A schematic diagram of the structure of an execution device provided in an embodiment of the present application;

[0065] Figure 8 A schematic diagram of the structure of a training device provided in an embodiment of the present application;

[0066] Fig. 9 A schematic diagram of the structure of a chip provided in an embodiment of the present application. DETAILED DESCRIPTION

[0067] The embodiments of the present invention are described below in conjunction with the accompanying drawings in the embodiments of the present invention. The terms used in the implementation mode of the present invention are only used to explain the specific embodiments of the present invention, and are not intended to limit the present invention.

[0068] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0069] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0070] First, the overall workflow of the artificial intelligence system is described. Figure 1 , Figure 1 The figure shows a structural diagram of the main framework of artificial intelligence. The following is an explanation of the above artificial intelligence theme framework from two dimensions: "intelligent information chain" (horizontal axis) and "IT value chain" (vertical axis). Among them, the "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it can be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, the data has undergone a condensation process of "data-information-knowledge-wisdom". The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry from the underlying infrastructure of human intelligence, information (providing and processing technology implementation) to the industrial ecological process of the system.

[0071] (1) Infrastructure

[0072] The infrastructure provides computing power support for the AI ​​system, enables communication with the outside world, and supports it through the basic platform. It communicates with the outside world through sensors; computing power is provided by smart chips (CPU, NPU, GPU, ASIC, FPGA and other hardware acceleration chips); the basic platform includes distributed computing frameworks and networks and other related platform guarantees and support, which can include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to obtain data, and these data are provided to the smart chips in the distributed computing system provided by the basic platform for calculation.

[0073] (2) Data

[0074] The data on the upper layer of the infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data of traditional devices, including business data of existing systems and perception data such as force, displacement, liquid level, temperature, and humidity.

[0075] (3) Data processing

[0076] Data processing usually includes data training, machine learning, deep learning, search, reasoning, decision-making and other methods.

[0077] Among them, machine learning and deep learning can symbolize and formalize data for intelligent information modeling, extraction, preprocessing, and training.

[0078] Reasoning refers to the process of simulating human intelligent reasoning in computers or intelligent systems, using formalized information to perform machine thinking and solve problems based on reasoning control strategies. Typical functions are search and matching.

[0079] Decision-making refers to the process of making decisions after intelligent information is reasoned, usually providing functions such as classification, sorting, and prediction.

[0080] (4) General ability

[0081] After the data has been processed as mentioned above, some general capabilities can be further formed based on the results of the data processing, such as an algorithm or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0082] (5) Smart products and industry applications

[0083] Smart products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of the overall artificial intelligence solution, which productizes intelligent information decision-making and realizes practical application. Its application areas mainly include: smart terminals, smart transportation, smart medical care, autonomous driving, safe cities, etc.

[0084] This application can be applied to fields such as safe cities in the field of artificial intelligence. The application scenarios in this application will be introduced below.

[0085] As the need to build a safe city becomes more and more urgent, video surveillance products are becoming more and more intelligent. Intelligent video analysis technology automatically analyzes video content through computers and mines valuable information from massive videos. In the past, people spent dozens of hours watching surveillance videos to complete the work, but with the help of intelligent video analysis systems, the results can be returned within a few seconds, greatly improving work efficiency.

[0086] In intelligent video analysis systems, pedestrian re-identification technology associates people with the same appearance under different cameras, thereby inferring the trajectory of pedestrians. A trajectory is a spatiotemporal route that specifies the order of time and place. Its input is a query image containing a pedestrian or other specified query conditions. The system retrieves all video clips containing the pedestrian or meeting the query conditions in a massive image library or video library, and then infers the pedestrian's trajectory based on the time and location information corresponding to the retrieval results.

[0087] Reference Figure 2 , Figure 2 This is a schematic diagram of an application architecture in an embodiment of the present application. Figure 2 As shown in , in a large data center, a large amount of labeled pedestrian re-identification (Re-identification, Re-ID) data can be used to train on a large server to obtain a deep model M, which can extract image features of the image and perform image similarity judgment based on the extracted image features. When the above-mentioned deep model M is deployed to the end side and used in a specific scene, a large amount of unlabeled pedestrian data can be obtained through the camera network. These unlabeled data can be used to adaptively train the above-mentioned deep model M on the server to obtain a deep model that adapts to the deployment scene. In addition, for the same scene, when the data changes, such as seasonal changes, the model can also be adaptively trained to update the model through the above method to obtain a deep model that adapts to the deployment scene.

[0088] Since the embodiments of the present application involve the application of a large number of neural networks, in order to facilitate understanding, the relevant terms and concepts of the neural networks that may be involved in the embodiments of the present application are first introduced below.

[0089] (1) Neural Network

[0090] A neural network may be composed of neural units, and a neural unit may refer to an operation unit with xs and intercept 1 as input, and the output of the operation unit may be:

[0091]

[0092] Where s=1, 2, ..., n, n is a natural number greater than 1, Ws is the weight of Xs, and b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolution layer, and the activation function can be a sigmoid function. A neural network is a network formed by connecting multiple single neural units mentioned above, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the characteristics of the local receptive field. The local receptive field can be an area composed of several neural units.

[0093] (2) Deep Neural Networks

[0094] A deep neural network (DNN), also known as a multi-layer neural network, can be understood as a neural network with multiple hidden layers. According to the position of different layers, the neural network inside the DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all hidden layers. The layers are fully connected, that is, any neuron in the i-th layer must be connected to any neuron in the i+1-th layer.

[0095] Although DNN looks complicated, the work of each layer is not complicated. In simple terms, it can be expressed as the following linear relationship: in, is the input vector, is the output vector, is the offset vector, W is the weight matrix (also called coefficient), and α() is the activation function. Each layer is just an input vector After such a simple operation, the output vector Since DNN has many layers, the coefficient W and the offset vector The number of these parameters is also relatively large. The definitions of these parameters in DNN are as follows: Take the coefficient W as an example: Assume that in a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as The superscript 3 represents the layer number of the coefficient W, while the subscripts correspond to the third layer index 2 of the output and the second layer index 4 of the input.

[0096] In summary, the coefficients from the kth neuron in the L-1th layer to the jth neuron in the Lth layer are defined as

[0097] It should be noted that the input layer does not have a W parameter. In a deep neural network, more hidden layers allow the network to better describe complex situations in the real world. Theoretically, the more parameters a model has, the higher its complexity and the greater its "capacity", which means it can complete more complex learning tasks. Training a deep neural network is the process of learning the weight matrix, and its ultimate goal is to obtain the weight matrix of all layers of the trained deep neural network (the weight matrix formed by many layers of vector W).

[0098] (3) Convolutional Neural Network

[0099] Convolutional neural network (CNN) is a deep neural network with a convolutional structure. Convolutional neural network contains a feature extractor consisting of a convolution layer and a subsampling layer, which can be regarded as a filter. Convolutional layer refers to the neuron layer in the convolutional neural network that performs convolution processing on the input signal. In the convolutional layer of the convolutional neural network, a neuron can only be connected to some neurons in the adjacent layers. A convolutional layer usually contains several feature planes, each of which can be composed of some rectangularly arranged neural units. The neural units in the same feature plane share weights, and the shared weights here are convolution kernels. Shared weights can be understood as the way to extract image information is independent of position. The convolution kernel can be initialized in the form of a matrix of random size, and the convolution kernel can obtain reasonable weights through learning during the training process of the convolutional neural network. In addition, the direct benefit of shared weights is to reduce the connection between the layers of the convolutional neural network, while reducing the risk of overfitting.

[0100] (4) Loss Function

[0101] In the process of training a deep neural network, because we hope that the output of the deep neural network is as close as possible to the value we really want to predict, we can compare the predicted value of the current network with the target value we really want, and then update the weight vector of each layer of the neural network according to the difference between the two (of course, there is usually an initialization process before the first update, that is, pre-configuring parameters for each layer in the deep neural network). For example, if the predicted value of the network is high, adjust the weight vector to make it predict a lower value, and keep adjusting until the deep neural network can predict the target value we really want or a value very close to the target value we really want. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value", which is the loss function or objective function, which are important equations used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so the training of the deep neural network becomes a process of minimizing this loss as much as possible.

[0102] (5) Back propagation algorithm

[0103] The neural network can use the error back propagation (BP) algorithm to correct the size of the parameters in the initial neural network model during the training process, so that the reconstruction error loss of the neural network model becomes smaller and smaller. Specifically, the forward transmission of the input signal to the output will generate error loss, and the parameters in the initial neural network model are updated by back propagating the error loss information, so that the error loss converges. The back propagation algorithm is a back propagation movement dominated by error loss, aiming to obtain the optimal parameters of the neural network model, such as the weight matrix.

[0104] Reference Figure 3 , Figure 3 An example of a model updating method provided in the present application is shown in FIG. Figure 3 As shown, a model updating method provided by an embodiment of the present application includes:

[0105] 301. Obtain a neural network model and multiple training images.

[0106] In an embodiment of the present application, the server can obtain a neural network model and multiple training images, wherein the neural network model can be a deep model trained by a cloud-side device on a large server based on a large amount of labeled Re-ID data.

[0107] In an embodiment of the present application, the server may acquire multiple training images, wherein the multiple training images may be a large number of unlabeled pedestrian images acquired through a camera network.

[0108] 302. Perform feature extraction on the multiple training images using the neural network model to obtain multiple target features.

[0109] In the embodiment of the present application, after the server obtains the neural network model and multiple training images, it can extract features from the multiple training images through the neural network model to obtain multiple target features. Figure 4 , Figure 4 A flow chart of a model updating method provided in an embodiment of the present application is shown as follows: Figure 4 As shown in , after the server obtains the neural network model and multiple training images, it can use the neural network model to extract features from the multiple training images to obtain multiple target features.

[0110] 303: Perform multiple first clustering processes on the multiple training images according to the multiple target features to obtain multiple first clustering results, each of the multiple first clustering results corresponds to a silhouette coefficient, and the silhouette coefficient represents clustering quality.

[0111] In the embodiment of the present application, the silhouette coefficient is related to the inter-class separation and the intra-class cohesion. The inter-class separation represents the distance between different cluster categories, and the intra-class cohesion represents the distance between different training images corresponding to the same cluster category.

[0112] In an embodiment of the present application, the silhouette coefficient is positively correlated with the distance between different clustering categories, and the silhouette coefficient is negatively correlated with the distance between different training images corresponding to the same clustering category. The multiple first clustering results include a second target clustering result and the first target clustering result, and the silhouette coefficient corresponding to the first target clustering result is greater than the silhouette coefficient corresponding to the second target clustering result.

[0113] In an embodiment of the present application, a gradual clustering method can be adopted. Specifically, the clustering process can be divided into two stages. In the first clustering stage, the server can perform multiple clustering processes on the multiple training images according to the multiple target features to obtain multiple first clustering results.

[0114] Specifically, in the first clustering stage, the server may perform multiple first clustering processes on the multiple training images according to the multiple target features, and update the parameters of the neural network based on the second loss function after each first clustering process.

[0115] Taking a clustering process in the first clustering stage as an example, the server can perform a clustering process on the multiple training images according to the multiple target features, and update the parameters of the neural network based on the second loss function to reduce the variance of the first similarity distribution between each training image in the multiple training images and the multiple clustering categories included in the first clustering result in the first clustering result obtained by the next first clustering process.

[0116] In the early stage of the first clustering stage, due to incomplete clustering, the same target (such as pedestrian) may belong to different clustering categories in the first clustering result. This effect can be reduced based on the soft multi-label learning strategy, where the loss function can be shown as follows:

[0117]

[0118] Among them, L(x i ) is the multi-label set matched by the training image i is the weight, v i is the target feature corresponding to the training image i, and C is the number of clustering categories in the clustering result. In each iteration, A classes with the highest similarity to the training image i can be selected, where A can be the preset number of clustering categories. At the same time, in order to highlight the role of high similarity classes, similarity is used as the weight α j In the embodiment of the present application, as the number of clusters decreases, soft multi-label learning can gradually degenerate into single-label learning as the data changes.

[0119] In the embodiment of the present application, in the early stage of the first clustering stage, due to incomplete clustering, in the obtained first clustering result, there may be a problem that the same training image belongs to different clustering categories (there is a certain similarity between the same training image and multiple clustering categories, so that the same training image belongs to different clustering categories), however, among multiple clustering categories, the training image may only have a large similarity with a part of the clustering categories, and the rest are small. After the neural network model is updated based on the above-mentioned second loss function, the similarity between the training image and a part of the clustering categories gradually increases, and the similarity between the training image and another part of the clustering categories gradually decreases, so that the variance of the similarity distribution between the training image and a part of the clustering categories included in the first clustering result gradually decreases, and the clustering categories with a large similarity to the training image gradually gather on a part of the clustering categories included in the first clustering result. As the first stage clustering process proceeds, soft multi-label learning can gradually degenerate into single label learning as the data changes.

[0120] In an embodiment of the present application, after each clustering process, the neural network model can be updated based on the above loss function, and the updated neural network model can be used to extract features of the training image to obtain multiple target features, and the next clustering process can be performed on multiple training images based on the obtained multiple target features.

[0121] In the embodiment of the present application, the number of clusters can be gradually reduced according to a certain number with the number of iterations, and clusters with higher similarity are preferentially clustered into one category. In the embodiment of the present application, the silhouette coefficient can be used as a clustering evaluation index, and the silhouette coefficient expression can be shown as follows:

[0122]

[0123] Among them, b i is the inter-class separation, which represents the distance between different cluster categories.

[0124] For example, b i It can be the average distance of all samples from training image i to neighbor clusters (nearby cluster categories), denoted as b. The neighbor cluster can be understood as the cluster category closest to training image i. It should be noted that since it is unknown which cluster is the neighbor cluster, it is necessary to traverse other clusters and select the cluster with the smallest average distance as the neighbor cluster.

[0125] a i It is the intra-class cohesion, which indicates the distance between different training images corresponding to the same clustering category.

[0126] For example, a i It can be the average distance between training image i and other samples of the same type.

[0127] In an embodiment of the present application, during the clustering process of the first stage, the number of clustering categories included in the first clustering result can be gradually reduced, and after each clustering process is completed, the corresponding silhouette coefficient can be calculated, and finally the optimal silhouette coefficient and the corresponding number of clustering categories M included in the first clustering result can be obtained.

[0128] In an embodiment of the present application, during the clustering processing of the first stage, multiple clustering processing may include an i-th clustering processing and a j-th clustering processing. After the j-th clustering processing, the server may obtain a second similarity between each target feature among the multiple target features when performing the i-th clustering processing and the j-th clustering processing. The second similarity includes an image feature similarity between each target feature among the multiple target features and a spatiotemporal constraint weight, wherein the spatiotemporal constraint weight when performing the j-th clustering processing is greater than the spatiotemporal constraint weight when performing the i-th clustering processing, and the spatiotemporal constraint weight represents the influence of the spatiotemporal constraint on the second similarity between the target features. The i-th clustering processing and the j-th clustering processing are performed on the multiple training images according to the second similarity, wherein i and j are positive integers.

[0129] In an embodiment of the present application, in the first few clustering processes of multiple clustering processes, the influence of space-time constraints on the similarity between training images may be very large. For example, there are multiple training images that are taken by camera 1, and there are multiple training images that are taken by camera 2. The shooting areas of camera 1 and camera 2 overlap greatly. Therefore, there are certain space-time constraints between the training images taken by camera 1 and the training images taken by camera 2. In the first few clustering processes of multiple clustering processes in the first clustering stage, since there are certain space-time constraints between the training images taken by camera 1 and the training images taken by camera 2, the objects that may be clustered are largely the training images taken by camera 1 and camera 2.

[0130] In the embodiment of the present application, the space-time constraint can be understood as the space-time transfer probability.

[0131] In the embodiment of the present application, an adaptive adjustment process of spatiotemporal constraints is added. The degree of spatiotemporal constraints is adjusted according to the clustering situation of each first clustering result, and the spatiotemporal constraints are reduced in the first few clustering processes of multiple clustering processes. Taking the multiple clustering processes including the i-th clustering process and the j-th clustering process as an example, the j-th clustering process is performed after the i-th clustering process. When the i-th clustering process and the j-th clustering process are performed, the second similarity between each target feature in the multiple target features is obtained, and the second similarity includes the image feature similarity between each target feature in the multiple target features and the spatiotemporal constraint weight, wherein the spatiotemporal constraint weight when performing the j-th clustering process is greater than the spatiotemporal constraint weight when performing the i-th clustering process, and the spatiotemporal constraint weight represents the influence of the spatiotemporal constraint on the second similarity between the target features; the i-th clustering process and the j-th clustering process are performed on the multiple training images according to the second similarity.

[0132] In an embodiment of the present application, the server can obtain a second similarity between each target feature in the multiple target features, wherein the second similarity includes an image feature similarity and a spatiotemporal constraint weight between each target feature in the multiple target features; the multiple training images are clustered multiple times according to the second similarity, and after each clustering process, the spatiotemporal constraint weights between each target feature in the multiple target features are updated according to the corresponding first clustering result.

[0133] Specifically, the degree of spatiotemporal constraints in the clustering process of the first stage can be updated based on the following formula:

[0134] s(x i ,x j )=s v (x i ,x j )*(1-α i *(P ST (x i ,x j )-1));

[0135] Among them, s v (x i ,x j ) is the similarity based on image features, P ST (x i ,x j ) is the spatiotemporal transfer probability of the image, α i is the constraint degree obtained according to the clustering situation, s(x i ,x j ) is used as the final similarity for clustering.

[0136] 304. Determine a first target clustering result from the multiple first clustering results according to multiple silhouette coefficients corresponding to the multiple first clustering results, where the first target clustering result includes M clustering categories.

[0137] In an embodiment of the present application, during the clustering process of the first stage, the number of clustering categories included in the first clustering result can be gradually reduced, and after each clustering process is completed, the corresponding silhouette coefficient can be calculated, and finally the optimal silhouette coefficient and the corresponding number of clustering categories M included in the first clustering result can be obtained.

[0138] In the embodiment of the present application, the silhouette coefficient is positively correlated with the distance between different clustering categories, and the silhouette coefficient is negatively correlated with the distance between different training images corresponding to the same clustering category, and the multiple first clustering results include the second target clustering result and the first target clustering result, and the silhouette coefficient corresponding to the first target clustering result is greater than the silhouette coefficient corresponding to the second target clustering result. That is, in the embodiment of the present application, the maximum silhouette coefficient can be selected as the optimal silhouette coefficient, and the number of clustering categories M included in the corresponding first clustering result can be determined.

[0139] 305. Perform a second clustering process on the multiple training images according to the multiple target features to obtain a second clustering result, and update the parameters of the neural network according to the first loss function, wherein the second clustering result includes the M number of clustering categories.

[0140] In an embodiment of the present application, after a first target clustering result is determined from the multiple first clustering results based on the multiple silhouette coefficients corresponding to the multiple first clustering results, and the first target clustering result includes M clustering categories, a second clustering process can be performed on the multiple training images based on the multiple target features to obtain a second clustering result, and the parameters of the neural network are updated according to the first loss function, and the number of clustering categories included in the second clustering result is M.

[0141] In the clustering process of the second stage, the neural network model updated after the last clustering process in the clustering process of the first stage can be used to extract features from multiple training images to obtain multiple target features, cluster the multiple training images according to the multiple target features to obtain a second clustering result, and update the parameters of the neural network according to the first loss function, and the number of clustering categories included in the second clustering result is M. That is, in the clustering process of the second stage, the number of clustering categories included in the second clustering result obtained each time is the number of clustering categories M included in the first clustering result corresponding to the optimal silhouette coefficient determined in the clustering process of the first stage.

[0142] In the prior art, in the process of training with unlabeled sample pairs of the target scene, each sample is initially regarded as a separate cluster category, and each iteration reduces a certain number of cluster categories according to the minimum distance principle. As the iteration proceeds, the number of cluster categories decreases, and the model gradually improves with the training performance. Finally, the models corresponding to the different numbers of cluster categories obtained are verified to determine the model with the best performance. In the above verification process, it is necessary to verify the models corresponding to the different numbers of cluster categories obtained based on labeled samples. However, in some scenarios, labeled samples are often difficult to obtain. In the embodiment of the present application, compared with the above-mentioned prior art, it is not necessary to verify the models corresponding to the different numbers of cluster categories obtained based on labeled samples to determine the optimal number of clusters.

[0143] In an embodiment of the present application, the first loss function includes a ternary loss function.

[0144] In an embodiment of the present application, the server can update the parameters of the neural network according to the ternary loss function and the second loss function to reduce the variance of the first similarity distribution between each of the multiple training images and the multiple clustering categories included in the second clustering result obtained by the next clustering processing, and increase the distance between the different clustering categories in the second clustering result obtained by the next clustering processing.

[0145] In the embodiment of the present application, the server may use the following triple loss function to update the neural network model to further enhance the feature discrimination capability:

[0146]

[0147] in represents the target feature of training image i, represents the target features of the training images of the same type as training image i, The target features of the training images of different classes from the training image i are represented. The ultimate goal of the triplet loss function is to make the distance between the training images of different classes greater than the distance between the training images of the same class, thereby improving the feature discrimination ability of the neural network model. The specific description of the second loss function can refer to the description in the above embodiment, which will not be repeated here.

[0148] The embodiment of the present application provides a model updating method, the method comprising: obtaining a neural network model and a plurality of training images; performing feature extraction on the plurality of training images through the neural network model to obtain a plurality of target features; performing a plurality of first clustering processes on the plurality of training images according to the plurality of target features to obtain a plurality of first clustering results, each of the plurality of first clustering results corresponding to a silhouette coefficient, the silhouette coefficient indicating the clustering quality; determining a first target clustering result from the plurality of first clustering results according to the plurality of silhouette coefficients corresponding to the plurality of first clustering results, the first target clustering result including M clustering categories; performing a second clustering process on the plurality of training images according to the plurality of target features to obtain a second clustering result, and updating the parameters of the neural network according to the first loss function, the number of clustering categories included in the second clustering result being the M. In the above manner, it is not necessary to determine the optimal number of clusters by verifying the models corresponding to the obtained numbers of different clustering categories based on labeled samples, and the optimal number of clustering categories can be directly selected by the result of the first clustering.

[0149] The following introduces a system architecture provided by an embodiment of the present application.

[0150] See also Figure 5 , an embodiment of the present application provides a system architecture 100. As shown in the system architecture 100, the data acquisition device 160 is used to collect training data. In the embodiment of the present application, the training data includes: multiple training images and a neural network model; and the training data is stored in the database 130, and the training device 120 trains and updates the neural network model based on the training data maintained in the database 130. It should be noted that in actual applications, the training data maintained in the database 130 does not necessarily all come from the collection of the data acquisition device 160, and may also be received from other devices. It should also be noted that the training device 120 does not necessarily train the neural network model entirely based on the training data maintained in the database 130, and may also obtain training data from the cloud or other places for model training. The above description should not be used as a limitation on the embodiments of the present application.

[0151] The target model / rule obtained by training the training device 120 can be applied to different systems or devices, such as Figure 5 The execution device 110 shown in the figure may be a terminal, such as a mobile phone terminal, a tablet computer, a laptop computer, an augmented reality (AR) / virtual reality (VR), a vehicle terminal, etc., or a server or a cloud. Figure 5In the embodiment, the execution device 110 is configured with an input / output (I / O) interface 112 for data interaction with an external device. A user may input data to the I / O interface 112 through a client device 140. The input data may include: multiple training images and a neural network model in the embodiment of the present application.

[0152] When the execution device 120 preprocesses the input data, or when the computing module 111 of the execution device 120 performs calculation and other related processing (such as implementing the function of the neural network in the present application), the execution device 120 can call the data, code, etc. in the data storage system 150 for the corresponding processing, and can also store the data, instructions, etc. obtained by the corresponding processing in the data storage system 150.

[0153] Finally, the I / O interface 112 returns the processing result to the client device 140 so as to provide it to the user.

[0154] It is worth noting that the training device 120 can generate corresponding target models / rules based on different training data for different goals or different tasks, and the corresponding target models / rules can be used to achieve the above goals or complete the above tasks, thereby providing users with the desired results.

[0155] exist Figure 5 In the case shown in the figure, the user can manually give input data, and the manual giving can be operated through the interface provided by the I / O interface 112. In another case, the client device 140 can automatically send input data to the I / O interface 112. If the client device 140 is required to automatically send input data and needs to obtain the user's authorization, the user can set the corresponding authority in the client device 140. The user can view the results output by the execution device 110 on the client device 140, and the specific presentation form can be a specific method such as display, sound, action, etc. The client device 140 can also be used as a data acquisition terminal to collect the input data of the input I / O interface 112 and the output results of the output I / O interface 112 as shown in the figure as new sample data, and store them in the database 130. Of course, it is also possible not to collect through the client device 140, but the I / O interface 112 directly stores the input data of the input I / O interface 112 and the output results of the output I / O interface 112 as new sample data in the database 130.

[0156] It is worth noting that Figure 5 This is only a schematic diagram of a system architecture provided by an embodiment of the present application. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, in Figure 5In the embodiment, the data storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 may also be placed in the execution device 110.

[0157] exist Figures 1 to 5 On the basis of the corresponding embodiments, in order to better implement the above solutions of the embodiments of the present application, the following also provides related devices for implementing the above solutions. Figure 6 , Figure 6 A schematic diagram of a structure of a model updating device 600 provided in an embodiment of the present application. The model updating device 600 may be a terminal device or a server. The model updating device 600 includes:

[0158] An acquisition module 601 is used to acquire a neural network model and a plurality of training images;

[0159] A feature extraction module 602 is used to extract features from the plurality of training images using the neural network model to obtain a plurality of target features;

[0160] A first clustering module 603 is used to perform a plurality of first clustering processes on the plurality of training images according to the plurality of target features to obtain a plurality of first clustering results, each of the plurality of first clustering results corresponding to a silhouette coefficient, and the silhouette coefficient represents clustering quality;

[0161] A determination module 604 is configured to determine a first target clustering result from the plurality of first clustering results according to a plurality of silhouette coefficients corresponding to the plurality of first clustering results, wherein the first target clustering result includes M clustering categories;

[0162] The second clustering module 605 is used to perform a second clustering process on the multiple training images according to the multiple target features to obtain a second clustering result, and update the parameters of the neural network according to the first loss function. The number of clustering categories included in the second clustering result is M.

[0163] Optionally, the silhouette coefficient is calculated based on inter-class separation and intra-class cohesion, the inter-class separation represents the distance between different cluster categories, and the intra-class cohesion represents the distance between different training images corresponding to the same cluster category.

[0164] Optionally, a silhouette coefficient corresponding to the first target clustering result is greater than silhouette coefficients corresponding to other clustering results among the multiple first clustering results except the first target clustering result.

[0165] Optionally, the first clustering module is specifically used to:

[0166] The plurality of training images are subjected to multiple first clustering processes according to the plurality of target features, and the parameters of the neural network are updated based on a second loss function after each first clustering process.

[0167] Optionally, the first clustering module is specifically used to:

[0168] After each first clustering process, the parameters of the neural network are updated based on the second loss function to reduce the variance of the first similarity distribution between each training image of the multiple training images and the multiple clustering categories included in the first clustering result in the first clustering result obtained by the next first clustering process.

[0169] Optionally, the multiple first clustering processes include an i-th clustering process and a j-th clustering process, the j-th clustering process is performed after the i-th clustering process, and the first clustering module is specifically configured to:

[0170] When performing the i-th clustering process and the j-th clustering process, obtaining a second similarity between each target feature in the multiple target features, the second similarity including an image feature similarity between each target feature in the multiple target features and a spatiotemporal constraint weight, wherein the spatiotemporal constraint weight when performing the j-th clustering process is greater than the spatiotemporal constraint weight when performing the i-th clustering process, and the spatiotemporal constraint weight represents the influence of the spatiotemporal constraint on the second similarity between the target features;

[0171] The i-th clustering process and the j-th clustering process are performed on the multiple training images according to the second similarity, wherein i and j are positive integers.

[0172] Optionally, the first clustering module is specifically used to:

[0173] Acquire a second similarity between each target feature in the plurality of target features, wherein the second similarity includes an image feature similarity and a spatiotemporal constraint weight between each target feature in the plurality of target features;

[0174] The plurality of training images are subjected to a plurality of first clustering processes according to the second similarity, and after each first clustering process, the spatiotemporal constraint weights between each target feature in the plurality of target features are updated according to the corresponding first clustering result.

[0175] Optionally, the second loss function comprises a ternary loss function.

[0176] Optionally, the second clustering module is specifically used to:

[0177] The parameters of the neural network are updated according to the ternary loss function and the second loss function to reduce the variance of the first similarity distribution between each of the multiple training images and the multiple clustering categories included in the second clustering result obtained by the next second clustering processing, and to increase the distance between the different clustering categories in the second clustering result obtained by the next clustering processing.

[0178] The embodiment of the present application provides a model updating device, wherein an acquisition module 601 acquires a neural network model and multiple training images; a feature extraction module 602 performs feature extraction on the multiple training images through the neural network model to obtain multiple target features; a first clustering module 603 performs multiple first clustering processes on the multiple training images according to the multiple target features to obtain multiple first clustering results, each of the multiple first clustering results corresponds to a silhouette coefficient, and the silhouette coefficient represents the clustering quality; a determination module 604 determines a first target clustering result from the multiple first clustering results according to the multiple silhouette coefficients corresponding to the multiple first clustering results, and the first target clustering result includes M clustering categories; a second clustering module 605 performs a second clustering process on the multiple training images according to the multiple target features to obtain a second clustering result, and updates the parameters of the neural network according to the first loss function, and the number of clustering categories included in the second clustering result is M. In the above manner, there is no need to verify the models corresponding to different numbers of cluster categories based on labeled samples to determine the optimal number of clusters. The optimal number of cluster categories can be directly selected based on the result of the first clustering.

[0179] Next, an execution device provided by an embodiment of the present application is introduced. Figure 7 , Figure 7 This is a schematic diagram of the structure of the execution device provided in the embodiment of the present application. The execution device 700 can be specifically manifested as a virtual reality VR device, a mobile phone, a tablet, a laptop, a smart wearable device, a monitoring data processing device, etc., which is not limited here. Specifically, the execution device 700 includes: a receiver 701, a transmitter 702, a processor 703 and a memory 704 (wherein the number of processors 703 in the execution device 700 can be one or more, Figure 7 In the example of FIG. 7 , the processor 703 may include an application processor 7031 and a communication processor 7032. In some embodiments of the present application, the receiver 701, the transmitter 702, the processor 703 and the memory 704 may be connected via a bus or other means.

[0180] The memory 704 may include a read-only memory and a random access memory, and provides instructions and data to the processor 703. A portion of the memory 704 may also include a non-volatile random access memory (NVRAM). The memory 704 stores processor and operation instructions, executable modules or data structures, or subsets thereof, or extended sets thereof, wherein the operation instructions may include various operation instructions for implementing various operations.

[0181] The processor 703 controls the operation of the execution device. In a specific application, the various components of the execution device are coupled together through a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus, and a status signal bus, etc. However, for the sake of clarity, various buses are referred to as bus systems in the figure.

[0182] The method disclosed in the above embodiment of the present application can be applied to the processor 703, or implemented by the processor 703. The processor 703 can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 703. The above processor 703 can be a general processor, a digital signal processor (digital signal processing, DSP), a microprocessor or a microcontroller, and can further include an application specific integrated circuit (application specific integrated circuit, ASIC), a field programmable gate array (field-programmable gate array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The processor 703 can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to execute, or a combination of hardware and software modules in the decoding processor to execute. The software module may be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 704, and the processor 703 reads the information in the memory 704 and completes the steps of the above method in combination with its hardware.

[0183] The receiver 701 can be used to receive input digital or character information and generate signal input related to the relevant settings and function control of the execution device. The transmitter 702 can be used to output digital or character information through the first interface; the transmitter 702 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; the transmitter 702 can also include a display device such as a display screen.

[0184] In one embodiment of the present application, the processor 703 is configured to execute Figure 3 The model updating method executed by the execution device in the corresponding embodiment. Specifically, the application processor 7031 can be based on Figure 3 The updated neural network model is used to process image data.

[0185] The present application also provides a training device. Figure 8 , Figure 8 is a structural diagram of a training device provided in an embodiment of the present application. The training device 800 may be deployed with Figure 6 The model updating device described in the corresponding embodiment is used to implement Figure 6 The function of the model updating device in the corresponding embodiment, specifically, the training device 800 is implemented by one or more servers, and the training device 800 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 822 (for example, one or more processors) and memory 832, one or more storage media 830 (for example, one or more mass storage devices) storing application programs 842 or data 844. Among them, the memory 832 and the storage medium 830 can be short-term storage or permanent storage. The program stored in the storage medium 830 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the training device. Furthermore, the central processing unit 822 can be configured to communicate with the storage medium 830 to execute a series of instruction operations in the storage medium 830 on the training device 800.

[0186] The training device 800 may also include one or more power supplies 826, one or more wired or wireless network interfaces 850, one or more input and output interfaces 858, and / or, one or more operating systems 841, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0187] In the embodiment of the present application, the central processing unit 822 is used to execute Figure 6The model updating method executed by the model updating device in the corresponding embodiment. Specifically, the central processing unit 822 is used to

[0188] Get a neural network model and multiple training images;

[0189] Performing feature extraction on the plurality of training images by using the neural network model to obtain a plurality of target features;

[0190] Performing a plurality of first clustering processes on the plurality of training images according to the plurality of target features to obtain a plurality of first clustering results, wherein each of the plurality of first clustering results corresponds to a silhouette coefficient, and the silhouette coefficient represents clustering quality;

[0191] Determining a first target clustering result from the plurality of first clustering results according to a plurality of silhouette coefficients corresponding to the plurality of first clustering results, wherein the first target clustering result includes M clustering categories;

[0192] A second clustering process is performed on the multiple training images according to the multiple target features to obtain a second clustering result, and the parameters of the neural network are updated according to the first loss function, and the number of clustering categories included in the second clustering result is M.

[0193] Optionally, the silhouette coefficient is calculated based on inter-class separation and intra-class cohesion, the inter-class separation represents the distance between different cluster categories, and the intra-class cohesion represents the distance between different training images corresponding to the same cluster category.

[0194] Optionally, a silhouette coefficient corresponding to the first target clustering result is greater than silhouette coefficients corresponding to other clustering results among the multiple first clustering results except the first target clustering result.

[0195] Optionally, performing a plurality of first clustering processes on the plurality of training images according to the plurality of target features comprises:

[0196] The plurality of training images are subjected to multiple first clustering processes according to the plurality of target features, and the parameters of the neural network are updated based on a second loss function after each first clustering process.

[0197] Optionally, updating the parameters of the neural network based on the second loss function after each first clustering process includes:

[0198] After each first clustering process, the parameters of the neural network are updated based on the second loss function to reduce the variance of the first similarity distribution between each training image of the multiple training images and the multiple clustering categories included in the first clustering result in the first clustering result obtained by the next first clustering process.

[0199] Optionally, the multiple first clustering processes include an i-th clustering process and a j-th clustering process, the j-th clustering process is performed after the i-th clustering process, and the performing the multiple first clustering processes on the multiple training images according to the multiple target features includes:

[0200] When performing the i-th clustering process and the j-th clustering process, obtaining a second similarity between each target feature in the multiple target features, the second similarity including an image feature similarity between each target feature in the multiple target features and a spatiotemporal constraint weight, wherein the spatiotemporal constraint weight when performing the j-th clustering process is greater than the spatiotemporal constraint weight when performing the i-th clustering process, and the spatiotemporal constraint weight represents the influence of the spatiotemporal constraint on the second similarity between the target features;

[0201] The i-th clustering process and the j-th clustering process are performed on the multiple training images according to the second similarity, wherein i and j are positive integers.

[0202] Optionally, performing a plurality of first clustering processes on the plurality of training images according to the plurality of target features comprises:

[0203] Acquire a second similarity between each target feature in the plurality of target features, wherein the second similarity includes an image feature similarity and a spatiotemporal constraint weight between each target feature in the plurality of target features;

[0204] The plurality of training images are subjected to a plurality of first clustering processes according to the second similarity, and after each first clustering process, the spatiotemporal constraint weights between each target feature in the plurality of target features are updated according to the corresponding first clustering result.

[0205] Optionally, the second loss function comprises a ternary loss function.

[0206] Optionally, updating the parameters of the neural network according to the first loss function includes:

[0207] The parameters of the neural network are updated according to the ternary loss function and the second loss function to reduce the variance of the first similarity distribution between each of the multiple training images and the multiple clustering categories included in the second clustering result obtained by the next second clustering processing, and to increase the distance between the different clustering categories in the second clustering result obtained by the next clustering processing.

[0208] The embodiment of the present application provides a model updating method, the method comprising: obtaining a neural network model and a plurality of training images; performing feature extraction on the plurality of training images through the neural network model to obtain a plurality of target features; performing a plurality of first clustering processes on the plurality of training images according to the plurality of target features to obtain a plurality of first clustering results, each of the plurality of first clustering results corresponding to a silhouette coefficient, the silhouette coefficient indicating the clustering quality; determining a first target clustering result from the plurality of first clustering results according to the plurality of silhouette coefficients corresponding to the plurality of first clustering results, the first target clustering result including M clustering categories; performing a second clustering process on the plurality of training images according to the plurality of target features to obtain a second clustering result, and updating the parameters of the neural network according to the first loss function, the number of clustering categories included in the second clustering result being the M. In the above manner, it is not necessary to determine the optimal number of clusters by verifying the models corresponding to the obtained numbers of different clustering categories based on labeled samples, and the optimal number of clustering categories can be directly selected by the result of the first clustering.

[0209] The present application also provides a computer program product which, when executed on a computer, enables the computer to execute the above Figure 7 The method described in the embodiment shown in the figure executes the steps executed by the device, or causes the computer to execute the above-mentioned Figure 8 The illustrated embodiment describes the steps performed by the training device in the method.

[0210] The present application also provides a computer-readable storage medium in which a program for signal processing is stored. When the program is run on a computer, the computer executes the above-mentioned Figure 7 The method described in the embodiment shown in the figure executes the steps executed by the device, or causes the computer to execute the above-mentioned Figure 8 The illustrated embodiment describes the steps performed by the training device in the method.

[0211] The execution device, training device or terminal device provided in the embodiment of the present application may be a chip, which includes: a processing unit and a communication unit. The processing unit may be a processor, and the communication unit may be an input / output interface, a pin or a circuit. The processing unit may execute the computer execution instructions stored in the storage unit, so that the chip in the execution device executes the above Figure 3 The model updating method described in the embodiment shown in the figure, or, to enable the chip in the training device to execute the above Figure 3The model updating method described in the illustrated embodiment. Optionally, the storage unit is a storage unit in the chip, such as a register, a cache, etc. The storage unit may also be a storage unit located outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.

[0212] For details, please refer to Fig. 9 , Fig. 9 A schematic diagram of the structure of a chip provided in an embodiment of the present application, the chip can be expressed as a neural network processor NPU 2000, NPU 2000 is mounted on the host CPU (Host CPU) as a coprocessor, and the host CPU assigns tasks. The core part of the NPU is the operation circuit 2003, which is controlled by the controller 2004 to extract matrix data from the memory and perform multiplication operations.

[0213] In some implementations, the operation circuit 2003 includes multiple processing units (Process Engine, PE) inside. In some implementations, the operation circuit 2003 is a two-dimensional systolic array. The operation circuit 2003 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 2003 is a general-purpose matrix processor.

[0214] For example, assume there is an input matrix A, a weight matrix B, and an output matrix C. The operation circuit takes the corresponding data of matrix B from the weight memory 2002 and caches it on each PE in the operation circuit. The operation circuit takes the matrix A data from the input memory 2001 and performs matrix operation with matrix B, and the partial result or final result of the matrix is ​​stored in the accumulator 2008.

[0215] The unified memory 2006 is used to store input data and output data. The weight data is directly transferred to the weight memory 2002 through the direct memory access controller (DMAC) 2005. The input data is also transferred to the unified memory 2006 through the DMAC.

[0216] BIU stands for Bus Interface Unit 2010 , which is used for the interaction between the AXI bus and the DMAC and the instruction fetch buffer (IFB) 2009 .

[0217] The bus interface unit 2010 (Bus Interface Unit, BIU for short) is used for the instruction fetch memory 2009 to obtain instructions from the external memory, and is also used for the storage unit access controller 2005 to obtain the original data of the input matrix A or the weight matrix B from the external memory.

[0218] DMAC is mainly used to transfer input data in the external memory DDR to the unified memory 2006 or to transfer weight data to the weight memory 2002 or to transfer input data to the input memory 2001.

[0219] The vector calculation unit 2007 includes multiple operation processing units, which further process the output of the operation circuit when necessary, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for non-convolutional / fully connected layer network calculations in neural networks, such as Batch Normalization, pixel-level summation, upsampling of feature planes, etc.

[0220] In some implementations, the vector calculation unit 2007 can store the processed output vector to the unified memory 2006. For example, the vector calculation unit 2007 can apply a linear function and / or a nonlinear function to the output of the operation circuit 2003, such as linear interpolation of the feature plane extracted by the convolution layer, and then, for example, a vector of accumulated values ​​to generate an activation value. In some implementations, the vector calculation unit 2007 generates a normalized value, a pixel-level summed value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 2003, such as for use in a subsequent layer in a neural network.

[0221] An instruction fetch buffer 2009 connected to the controller 2004, for storing instructions used by the controller 2004;

[0222] The unified memory 2006, the input memory 2001, the weight memory 2002 and the instruction fetch memory 2009 are all on-chip memories. The external memory is private to the NPU hardware architecture.

[0223] The processor mentioned in any of the above may be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits for controlling the execution of:

[0224] Get a neural network model and multiple training images;

[0225] Performing feature extraction on the plurality of training images by using the neural network model to obtain a plurality of target features;

[0226] Performing a plurality of first clustering processes on the plurality of training images according to the plurality of target features to obtain a plurality of first clustering results, wherein each of the plurality of first clustering results corresponds to a silhouette coefficient, and the silhouette coefficient represents clustering quality;

[0227] Determining a first target clustering result from the plurality of first clustering results according to a plurality of silhouette coefficients corresponding to the plurality of first clustering results, wherein the first target clustering result includes M clustering categories;

[0228] A second clustering process is performed on the multiple training images according to the multiple target features to obtain a second clustering result, and the parameters of the neural network are updated according to the first loss function, and the number of clustering categories included in the second clustering result is M.

[0229] Optionally, the silhouette coefficient is calculated based on inter-class separation and intra-class cohesion, the inter-class separation represents the distance between different cluster categories, and the intra-class cohesion represents the distance between different training images corresponding to the same cluster category.

[0230] Optionally, a silhouette coefficient corresponding to the first target clustering result is greater than silhouette coefficients corresponding to other clustering results among the multiple first clustering results except the first target clustering result.

[0231] Optionally, performing a plurality of first clustering processes on the plurality of training images according to the plurality of target features comprises:

[0232] The plurality of training images are subjected to multiple first clustering processes according to the plurality of target features, and the parameters of the neural network are updated based on a second loss function after each first clustering process.

[0233] Optionally, updating the parameters of the neural network based on the second loss function after each first clustering process includes:

[0234] After each first clustering process, the parameters of the neural network are updated based on the second loss function to reduce the variance of the first similarity distribution between each training image of the multiple training images and the multiple clustering categories included in the first clustering result in the first clustering result obtained by the next first clustering process.

[0235] Optionally, the multiple first clustering processes include an i-th clustering process and a j-th clustering process, the j-th clustering process is performed after the i-th clustering process, and the performing the multiple first clustering processes on the multiple training images according to the multiple target features includes:

[0236] When performing the i-th clustering process and the j-th clustering process, obtaining a second similarity between each target feature in the multiple target features, the second similarity including an image feature similarity between each target feature in the multiple target features and a spatiotemporal constraint weight, wherein the spatiotemporal constraint weight when performing the j-th clustering process is greater than the spatiotemporal constraint weight when performing the i-th clustering process, and the spatiotemporal constraint weight represents the influence of the spatiotemporal constraint on the second similarity between the target features;

[0237] The i-th clustering process and the j-th clustering process are performed on the multiple training images according to the second similarity, wherein i and j are positive integers.

[0238] Optionally, performing a plurality of first clustering processes on the plurality of training images according to the plurality of target features comprises:

[0239] Acquire a second similarity between each target feature in the plurality of target features, wherein the second similarity includes an image feature similarity and a spatiotemporal constraint weight between each target feature in the plurality of target features;

[0240] The plurality of training images are subjected to a plurality of first clustering processes according to the second similarity, and after each first clustering process, the spatiotemporal constraint weights between each target feature in the plurality of target features are updated according to the corresponding first clustering result.

[0241] Optionally, the second loss function comprises a ternary loss function.

[0242] Optionally, updating the parameters of the neural network according to the first loss function includes:

[0243] The parameters of the neural network are updated according to the ternary loss function and the second loss function to reduce the variance of the first similarity distribution between each of the multiple training images and the multiple clustering categories included in the second clustering result obtained by the next second clustering processing, and to increase the distance between the different clustering categories in the second clustering result obtained by the next clustering processing.

[0244] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0245] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0246] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0247] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. A model updating method, characterized in that: The method comprises: Obtaining a neural network model and a plurality of training images, wherein the plurality of training images are unlabeled images; Performing feature extraction on the plurality of training images by using the neural network model to obtain a plurality of target features; Performing a plurality of first clustering processes on the plurality of training images according to the plurality of target features to obtain a plurality of first clustering results, each of the plurality of first clustering results corresponding to a silhouette coefficient, and the silhouette coefficient indicating clustering quality; determining a first target clustering result from the plurality of first clustering results according to the plurality of silhouette coefficients corresponding to the plurality of first clustering results, the first target clustering result including M clustering categories, and the first target clustering result being a result corresponding to the best silhouette coefficient among the plurality of first clustering results; Performing a second clustering process on the multiple training images according to the multiple target features to obtain a second clustering result, and updating the parameters of the neural network according to the first loss function, the second clustering result includes the M number of clustering categories.

2. The method according to claim 1, characterized in that The silhouette coefficient is calculated based on the inter-class separation and the intra-class cohesion. The inter-class separation represents the distance between different cluster categories, and the intra-class cohesion represents the distance between different training images corresponding to the same cluster category.

3. The method according to claim 1, characterized in that The silhouette coefficient corresponding to the first target clustering result is greater than the silhouette coefficients corresponding to other clustering results among the multiple first clustering results except the first target clustering result.

4. The method according to any one of claims 1 to 3, characterized in that: The performing a plurality of first clustering processes on the plurality of training images according to the plurality of target features comprises: The plurality of training images are subjected to multiple first clustering processes according to the plurality of target features, and the parameters of the neural network are updated based on a second loss function after each first clustering process.

5. The method according to claim 4, characterized in that The updating of the parameters of the neural network based on the second loss function after each first clustering process comprises: After each first clustering process, the parameters of the neural network are updated based on the second loss function to reduce the variance of the first similarity distribution between each training image of the multiple training images and the multiple clustering categories included in the first clustering result in the first clustering result obtained by the next first clustering process.

6. The method according to any one of claims 1 to 3, characterized in that: The multiple first clustering processes include an i-th clustering process and a j-th clustering process, wherein the j-th clustering process is performed after the i-th clustering process, and the multiple first clustering processes are performed on the multiple training images according to the multiple target features, including: When performing the i-th clustering process and the j-th clustering process, obtaining a second similarity between each target feature in the multiple target features, the second similarity including an image feature similarity between each target feature in the multiple target features and a spatiotemporal constraint weight, wherein the spatiotemporal constraint weight when performing the j-th clustering process is greater than the spatiotemporal constraint weight when performing the i-th clustering process, and the spatiotemporal constraint weight represents the influence of the spatiotemporal constraint on the second similarity between the target features; The i-th clustering process and the j-th clustering process are performed on the multiple training images according to the second similarity, wherein i and j are positive integers.

7. The method according to any one of claims 1 to 3, characterized in that: The performing a plurality of first clustering processes on the plurality of training images according to the plurality of target features comprises: Acquire a second similarity between each target feature in the plurality of target features, wherein the second similarity includes an image feature similarity and a spatiotemporal constraint weight between each target feature in the plurality of target features; The plurality of training images are subjected to a plurality of first clustering processes according to the second similarity, and after each first clustering process, the spatiotemporal constraint weights between each target feature in the plurality of target features are updated according to the corresponding first clustering result.

8. The method according to any one of claims 1 to 3, characterized in that: The first loss function includes a ternary loss function.

9. The method according to any one of claims 1 to 3, characterized in that: The updating of the parameters of the neural network according to the first loss function comprises: The parameters of the neural network are updated according to the ternary loss function and the second loss function to reduce the variance of the first similarity distribution between each training image of the multiple training images and the multiple clustering categories included in the second clustering result obtained by the next second clustering processing, and to increase the distance between different clustering categories in the second clustering result obtained by the next clustering processing.

10. A model updating device, characterized in that: The device comprises: An acquisition module, used to acquire a neural network model and a plurality of training images, wherein the plurality of training images are unlabeled images; A feature extraction module, used to extract features from the plurality of training images through the neural network model to obtain a plurality of target features; a first clustering module, configured to perform a plurality of first clustering processes on the plurality of training images according to the plurality of target features to obtain a plurality of first clustering results, wherein each of the plurality of first clustering results corresponds to a silhouette coefficient, and the silhouette coefficient represents clustering quality; A determination module, configured to determine a first target clustering result from the plurality of first clustering results according to a plurality of silhouette coefficients corresponding to the plurality of first clustering results, wherein the first target clustering result includes M clustering categories, and the first target clustering result is a result corresponding to the best silhouette coefficient among the plurality of first clustering results; A second clustering module is used to perform a second clustering process on the multiple training images according to the multiple target features to obtain a second clustering result, and to update the parameters of the neural network according to the first loss function, wherein the number of clustering categories included in the second clustering result is M.

11. The device according to claim 10, characterized in that The silhouette coefficient is calculated based on the inter-class separation and the intra-class cohesion. The inter-class separation represents the distance between different cluster categories, and the intra-class cohesion represents the distance between different training images corresponding to the same cluster category.

12. The device according to claim 10, characterized in that The silhouette coefficient corresponding to the first target clustering result is greater than the silhouette coefficients corresponding to other clustering results among the multiple first clustering results except the first target clustering result.

13. The device according to any one of claims 10 to 12, characterized in that: The first clustering module is specifically used for: The plurality of training images are subjected to multiple first clustering processes according to the plurality of target features, and the parameters of the neural network are updated based on a second loss function after each first clustering process.

14. The device according to claim 13, characterized in that The first clustering module is specifically used for: After each first clustering process, the parameters of the neural network are updated based on the second loss function to reduce the variance of the first similarity distribution between each training image of the multiple training images and the multiple clustering categories included in the first clustering result in the first clustering result obtained by the next first clustering process.

15. The device according to any one of claims 10 to 12, characterized in that: The multiple first clustering processes include an i-th clustering process and a j-th clustering process, wherein the j-th clustering process is performed after the i-th clustering process, and the first clustering module is specifically configured to: When performing the i-th clustering process and the j-th clustering process, obtaining a second similarity between each target feature in the multiple target features, the second similarity including an image feature similarity between each target feature in the multiple target features and a spatiotemporal constraint weight, wherein the spatiotemporal constraint weight when performing the j-th clustering process is greater than the spatiotemporal constraint weight when performing the i-th clustering process, and the spatiotemporal constraint weight represents the influence of the spatiotemporal constraint on the second similarity between the target features; The i-th clustering process and the j-th clustering process are performed on the multiple training images according to the second similarity, wherein i and j are positive integers.

16. The device according to any one of claims 10 to 12, characterized in that: The first clustering module is specifically used for: Acquire a second similarity between each target feature in the plurality of target features, wherein the second similarity includes an image feature similarity and a spatiotemporal constraint weight between each target feature in the plurality of target features; The plurality of training images are subjected to a plurality of first clustering processes according to the second similarity, and after each first clustering process, the spatiotemporal constraint weights between each target feature in the plurality of target features are updated according to the corresponding first clustering result.

17. The device according to any one of claims 10 to 12, characterized in that: The first loss function includes a ternary loss function.

18. The device according to any one of claims 10 to 12, characterized in that: The second clustering module is specifically used for: The parameters of the neural network are updated according to the ternary loss function and the second loss function to reduce the variance of the first similarity distribution between each training image of the multiple training images and the multiple clustering categories included in the second clustering result obtained by the next second clustering processing, and to increase the distance between different clustering categories in the second clustering result obtained by the next clustering processing.

19. A model updating device, characterized in that: The device comprises: a memory, a processor and a bus system, wherein the memory is used to store a program, and the processor is used to obtain the program in the memory to execute the model updating method according to any one of claims 1 to 9.

20. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed on a computer, enables the computer to execute the model updating method according to any one of claims 1 to 9.

21. A computer program product, characterized in that When the method is executed on a computer, the computer is enabled to execute the model updating method as claimed in any one of claims 1 to 9.

Images