Method and device for recognizing image
Through the collaborative evolution method of image feature extraction and cloud model training on the end-side device, the problem of limited recognition capabilities of the end-side device model is solved, and more efficient image recognition capabilities are achieved.
Patent Information
- Application Number
- CN202411104343.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-13
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2044-08-13
AI Technical Summary
The model recognition capabilities of the end-side device deployment are limited and cannot be directly trained using complex scenario data, resulting in untrustworthy products after training.
By inputting the images collected by the end-side device into the lightweight network model, the feature matrix is obtained and the general network model deployed in the cloud is trained, and then using the training results and image sets to train the lightweight network model to achieve the evolution of the lightweight network model , thereby improving the image recognition capability of the end-side device.
Through collaborative evolution, the recognition ability of the end-side device deployment model is improved, the problem of limited model recognition ability is solved, and the recognition accuracy and generalization ability are improved.
Smart Images

Figure CN118628894B_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present invention relate to the field of image recognition, and in particular, to a method and device for recognizing an image. Background Art
[0002] The end-cloud model is a model that combines cloud computing and end-side computing. In this model, computing tasks can be processed in the cloud or on the end-side device. By flexibly allocating tasks between the cloud and the end-side, more efficient computing and data processing can be achieved. The end-cloud model can fully utilize the powerful computing power and resources of cloud computing, while also reducing latency and bandwidth consumption and improving data processing efficiency. It has been widely used in fields that require large amounts of data processing and analysis, such as the Internet of Things, smart manufacturing, and smart transportation.
[0003] In actual applications, edge devices often need to face a large number of complex scenarios, but the training capabilities of edge devices are limited by the capacity limit of the model deployed on the edge devices. They cannot directly use complex scenario data for training, which is prone to risks caused by training divergence and data distribution deviation, resulting in unreliable training products.
[0004] There is currently no effective solution to the above problems. Summary of the invention
[0005] Embodiments of the present invention provide a method and apparatus for recognizing an image, so as to at least solve the problem of limited model recognition capability deployed by a terminal-side device in the related art.
[0006] According to an embodiment of the present invention, a method for recognizing an image is provided, comprising: inputting an image set collected by an end-side device into an i-th lightweight network model, obtaining a first feature matrix output by the i-th lightweight network model, wherein the i-th lightweight network model is a model obtained after the i-th training of the initial lightweight network model, wherein i is an integer greater than or equal to 1, and the lightweight network model is deployed on the end-side device; training the i-th general network model i+1 times through the first feature matrix to obtain a first image recognition result output by the i+1-th general network model, wherein the i-th general network model is a model obtained after the i-th training of the initial general network model, the i+1-th general network model is a model obtained after the i+1-th training of the i-th general network model, and the general network model is deployed in the cloud; training the i-th lightweight network model through the first image recognition result and the image set to obtain the i+1-th lightweight network model; recognizing a target object in the image collected by the end-side device through the i+1-th lightweight network model.
[0007] In an exemplary embodiment, the i-th general network model is trained i+1 times using the first feature matrix, including: inputting the first feature matrix into a feature mapping network, converting the first feature matrix into a third feature matrix through the feature mapping network, wherein the feature mapping network is located between the general network model and the lightweight network model; inputting the third feature matrix into a feature upgrade network of the i-th general network model, training the feature upgrade network using the third feature matrix, and obtaining a target feature upgrade network.
[0008] In an exemplary embodiment, after obtaining the target feature upgrading network, the method further includes: using the cloud image set to train the i-th feature extraction network and the target feature upgrading network to obtain the i+1-th general network model, wherein the cloud image set is the image set stored in the cloud, the general network model includes a feature extraction network and a feature upgrading network, the output of the feature extraction network is the input of the feature upgrading network, and the i-th feature extraction network is the feature extraction network in the i-th general network model.
[0009] In an exemplary embodiment, an image set collected by an end-side device is input into an i-th lightweight network model, and a first feature matrix output by the i-th lightweight network model is obtained, including: extracting features of images in the image set through the i-th lightweight network model to obtain a set of image feature vectors; dividing the image feature vector set into N image feature vector subsets, each of the image feature vector subsets including at least one image feature vector of the same sample object, the sample object is an object in the image collected by the end-side device, and N is an integer greater than or equal to 1; and obtaining the first feature matrix through the N image feature vector subsets.
[0010] In an exemplary embodiment, obtaining the first feature matrix through the N image feature vector subsets includes: screening the image feature vectors in each of the image feature vector subsets according to the signal-to-noise ratio of the image feature vector to obtain N target image feature vector subsets; and determining the N target image feature vector subsets as the first feature matrix.
[0011] In an exemplary embodiment, the image feature vectors in each of the image feature vector subsets are screened according to the signal-to-noise ratio of the image feature vector to obtain N target image feature vector subsets, including: performing the following operations on each image feature vector in the image feature vector subset, and determining the image feature vector performing the following operations as the current image feature vector: determining the signal-to-noise ratio of the current image feature vector; determining that the current image feature vector is valid if the signal-to-noise ratio is greater than or equal to a preset signal-to-noise ratio threshold; determining that the current image feature vector is invalid and deleting the current image feature vector if the signal-to-noise ratio is less than a preset signal-to-noise ratio threshold; and determining the valid image feature vector as the image feature vector in the target image feature vector subset.
[0012] In an exemplary embodiment, the i-th lightweight network model is trained through the first image recognition result and the image set to obtain the i+1-th lightweight network model, including: performing the j-th round of training on the i-th lightweight network model through the image set to obtain a second image recognition result output from the j-th round of training of the i-th lightweight network model, wherein j is an integer greater than or equal to 1; calculating the similarity between the second image recognition result and the first image recognition result to obtain a calculation result; when the calculation result is greater than or equal to a preset similarity threshold, determining the lightweight network model obtained by the j-th round of training as the i+1-th lightweight network model; when the calculation result is less than the preset similarity threshold, performing the j+1-th round of training on the i-th lightweight network model.
[0013] According to another embodiment of the present invention, a device for recognizing an image is provided, comprising: an acquisition module, used to input an image set collected by an end-side device into an i-th lightweight network model, and obtain a first feature matrix output by the i-th lightweight network model, wherein the i-th lightweight network model is a model obtained after the i-th training of the initial lightweight network model, wherein i is an integer greater than or equal to 1, and the lightweight network model is deployed on the end-side device; a first training module, used to train the i-th general network model i+1 times through the first feature matrix to obtain the i+1th general network model. The first image recognition result output by the model, wherein the i-th general network model is a model obtained after the initial general network model is trained for the i-th time, the i+1-th general network model is a model obtained after the i-th general network model is trained for the i+1-th time, and the general network model is deployed in the cloud; a second training module is used to train the i-th lightweight network model through the first image recognition result and the image set to obtain the i+1-th lightweight network model; an identification module is used to identify the target object in the image collected by the terminal side device through the i+1-th lightweight network model.
[0014] According to yet another embodiment of the present invention, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, wherein the computer program implements the steps of any of the above methods when executed by a processor.
[0015] According to yet another embodiment of the present invention, there is provided an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.
[0016] According to yet another embodiment of the present invention, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the steps of any of the above methods are implemented.
[0017] According to the present invention, an image set collected by a terminal device is input into an i-th lightweight network model to obtain a first characteristic matrix output by the i-th lightweight network model, wherein the i-th lightweight network model is a model obtained after the initial lightweight network model is trained for the i-th time, wherein i is an integer greater than or equal to 1, and the lightweight network model is deployed on the terminal device; the i-th general network model is trained for i+1 times through the first characteristic matrix to obtain a first image recognition result output by the i+1-th general network model, wherein the i-th general network model is a model obtained after the initial general network model is trained for the i-th time, the i+1-th general network model is a model obtained after the i-th general network model is trained for the i+1 time, and the general network model is deployed on the cloud; the i-th lightweight network model is trained through the first image recognition result and the image set to obtain the i+1-th lightweight network model; the target object in the image collected by the terminal device is recognized through the i+1-th lightweight network model.
[0018] The feature matrix extracted by the lightweight network model deployed on the end-side device is used to train the general network model deployed on the cloud, and the training results of the general network model and the feature matrix extracted by the lightweight network model are used to train the lightweight network model, so that the target object in the image is recognized by the trained lightweight network model, realizing the co-evolution of the end-cloud model. Therefore, the problem of limited recognition ability of the model deployed on the end-side device in the related technology can be solved, and the recognition ability of the model deployed on the end-side device can be improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] Figure 1 It is a hardware structure block diagram of a mobile terminal of a method for recognizing an image according to an embodiment of the present invention;
[0020] Figure 2 is a flowchart of a method for recognizing an image according to an embodiment of the present invention;
[0021] Figure 3 is a flow chart of feature vector merging according to an embodiment of the present invention;
[0022] Figure 4 is a schematic diagram of the structure of a general network model according to an embodiment of the present application;
[0023] Figure 5 is a training flow chart of a feature upgrading network according to an embodiment of the present invention;
[0024] Figure 6 is a training flow chart of a feature extraction network according to an embodiment of the present invention;
[0025] Figure 7 is a training flow chart of a lightweight network model according to an embodiment of the present invention;
[0026] Figure 8 is a specific flow chart of the evolution of the end-cloud model system according to an embodiment of the present invention;
[0027] Fig. 9 4 is a structural block diagram of an apparatus for recognizing an image according to an embodiment of the present invention. DETAILED DESCRIPTION
[0028] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings and in combination with the embodiments.
[0029] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.
[0030] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 FIG. 1 is a hardware structure block diagram of a mobile terminal of a method for recognizing an image according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.
[0031] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the method for recognizing an image in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the mobile terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0032] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.
[0033] In this embodiment, a method running on the above mobile terminal or network architecture is provided. Figure 2 is a flowchart of a method for recognizing an image according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:
[0034] Step S202: inputting an image set collected by the end-side device into an i-th lightweight network model, and obtaining a first feature matrix output by the i-th lightweight network model, wherein the i-th lightweight network model is a model obtained after the i-th training of the initial lightweight network model, wherein i is an integer greater than or equal to 1, and the lightweight network model is deployed on the end-side device;
[0035] The terminal device may be a camera, monitoring device, etc.; the image set may be a set of images collected by multiple terminal devices within a period of time; the first feature matrix may be a set of feature vectors output by a lightweight network model; the lightweight network model may be a model deployed on the terminal device, and the network model may be an expert model used to process and analyze data and provide real-time decision support or intelligent services. The first feature matrix is obtained by extracting features from the images in the image set collected by the terminal device through the lightweight network model.
[0036] Since the end-side expert model usually has lower latency and higher real-time performance, it is suitable for application scenarios that require fast response, such as industrial automation, intelligent transportation and other fields. Therefore, by deploying the expert model on the end-side device, the dependence on cloud resources can be reduced and the stability and security of the system can be improved. It should be noted that the expert model deployed on the end-side device can be a deep staggered network model (ResNet), a deep convolutional network model (GoogleLeNet), a visual model (ViT), etc., and there is no restriction here.
[0037] Specifically, features of the images in the image set are extracted through the i-th lightweight network model to obtain a set of image feature vectors; the set of image feature vectors is divided into N image feature vector subsets, each of the image feature vector subsets includes at least one image feature vector of the same sample object, the sample object is an object in the image collected by the end-side device, and N is an integer greater than or equal to 1; the first feature matrix is obtained through the N image feature vector subsets.
[0038] The above-mentioned image feature vector set division can be performed according to the spatiotemporal relationship of the terminal side device, and the spatiotemporal relationship can be the time when the terminal side device collects the image, the geographical location of the terminal side device, the spatial relationship between the terminal side devices, etc.; obtain a set of images collected by multiple terminal side devices within a period of time, extract features from the set in the image set through a lightweight network, and cluster the extracted feature vectors based on information such as image collection time, geographical location, regional information, and the spatial relationship between the two collection devices to obtain a first feature matrix.
[0039] Since multiple end-side devices differ in time and space, that is, in the same time period, the targets captured by the end-side devices at the same point are continuous, but the targets captured by the end-side devices at different points may have inconsistent identities. According to the relationship between the end-side and the ends (that is, the time-space information), the feature vectors are divided and archived to obtain the feature set of each sample object, and a feature file of the sample object is established. The time-space characteristics of a single end-side device and the correlation between multiple end-side devices are fully utilized, avoiding the multiple-file situation introduced by too dense end-side points. At the same time, due to the archiving process, the multiple-file situation of a single sample is eliminated to ensure the quality of end-side data.
[0040] For example Figure 3 is a flow chart of feature vector merging according to an embodiment of the present invention. Figure 3 As shown, images collected by multiple cameras within a period of time are obtained, and features of the collected images are extracted through a lightweight network model deployed on the terminal side to obtain multiple feature vectors; the multiple feature vectors extracted are clustered and combined according to information such as the time when the cameras collect the images, the geographical locations of the cameras, the regional information, and the spatial relationship between the two cameras to obtain a subset of image feature vectors, and finally a first feature matrix is obtained based on the subset of image feature vectors.
[0041] Optionally, obtaining the first feature matrix through the N image feature vector subsets includes: screening the image feature vectors in each of the image feature vector subsets according to the signal-to-noise ratio of the image feature vector to obtain N target image feature vector subsets; and determining the N target image feature vector subsets as the first feature matrix.
[0042] The signal-to-noise ratio can be the ratio of information to noise in each feature vector, wherein the larger the signal-to-noise ratio, the more information and the less noise in the feature vector; conversely, the smaller the signal-to-noise ratio, the less information and the more noise in the feature vector. The image feature vector is screened by the signal-to-noise ratio to determine the first feature matrix, thereby ensuring the validity of the first feature matrix, facilitating subsequent model training, and improving the accuracy and stability of the trained model.
[0043] Step S204, training the ith general network model for i+1 times using the first feature matrix to obtain a first image recognition result output by the i+1th general network model, wherein the ith general network model is a model obtained after the initial general network model is trained for the i-th time, the i+1th general network model is a model obtained after the ith general network model is trained for the i+1th time, and the general network model is deployed in the cloud;
[0044] The above general network model can be a model deployed in the cloud. Figure 4 is a schematic diagram of the structure of a general network model according to an embodiment of the present application, such as Figure 4 As shown, the general network model mainly includes two parts: feature extraction network and feature upgrading network. Among them, the feature extraction network is mainly used to map the image to the feature space. Its input is the image, and its output is the feature vector of the target in the image. The feature vector represents the identity of the image. The specific network structure can choose the open source basic model, such as ResNet, GoogleLeNet, ViT, etc.; the feature upgrading network is mainly used to optimize the feature distribution in the feature space. Its input is the output of the feature extraction network, and its output is the optimized new feature vector in the feature space. The specific network structure can be a cascade model of several fully connected layers, etc. It should be noted that the network structures of the above-mentioned feature extraction network and feature upgrading network are only possible examples, and no restrictions are made here.
[0045] By training the universal network model through the first feature matrix, a first image recognition result output by the universal network model can be obtained.
[0046] Specifically, the first feature matrix is input into a feature mapping network, and the first feature matrix is converted into a third feature matrix through the feature mapping network, wherein the feature mapping network is located between the general network model and the lightweight network model; the third feature matrix is input into a feature upgrade network of the i-th general network model, and the feature upgrade network is trained through the third feature matrix to obtain a target feature upgrade network.
[0047] The above-mentioned first feature matrix can be the feature extracted by the lightweight network model, which is the end space dimension feature; the above-mentioned third feature matrix can be the feature obtained after performing end-cloud feature mapping on the first feature matrix, which is the cloud space dimension feature; the above-mentioned feature mapping network can include an activation function and a fully connected layer, etc., and the activation function can be a sigmoid function, a tanh function, a relu function, etc.; the first feature matrix is subjected to end-cloud feature mapping through the feature mapping network to obtain the third feature matrix, thereby completing the feature space mapping from the end feature to the cloud feature, realizing the conversion of the spatial dimension, so that the feature matrix output by the lightweight network model is converted to meet the dimensional requirements of the feature matrix input by the general network model, and ensuring that the features extracted by the lightweight network model can be used for model training of the general network model deployed on the cloud.
[0048] For example, Figure 5 is a training flow chart of a feature upgrading network according to an embodiment of the present invention, such as Figure 5 As shown, the first feature matrix is input into the feature remapping module, and the feature space mapping of the end features to the cloud features is completed through the feature mapping network in the feature remapping module to obtain the third feature matrix. The third feature matrix is used as input to train the feature upgrade network in the general network model, and the optimized features are output to complete the metric learning of the feature upgrade network. At the same time, after the feature upgrade network converges, the model parameters of the feature upgrade network are fixed to obtain the target feature upgrade network.
[0049] The feature upgrade network in the general network model is trained through lightweight network model extraction, and then the training of the general network model is completed to achieve the upgrade of the general network model. The optimization of the general network model only depends on the terminal features instead of the image, which solves the dependence of the general network model on the images collected by the terminal device, protects the user's privacy, and realizes model evolution.
[0050] As an optional implementation, after obtaining the target feature upgrading network, the method further includes: using the cloud image set to train the i-th feature extraction network and the target feature upgrading network to obtain the i+1-th general network model, wherein the cloud image set is the image set stored in the cloud, the general network model includes a feature extraction network and a feature upgrading network, the output of the feature extraction network is the input of the feature upgrading network, and the i-th feature extraction network is the feature extraction network in the i-th general network model.
[0051] The above-mentioned cloud image collection can be an image collection stored in the cloud. The general network model is trained through the cloud image collection instead of directly using the images collected by the terminal device, thereby avoiding the privacy issues existing in the images collected by the terminal device and protecting the privacy of the user.
[0052] Figure 6 is a training flow chart of a feature extraction network according to an embodiment of the present invention, such as Figure 6 As shown, the universal basic network is trained through the image set stored in the cloud, wherein the feature upgrade network in the universal basic network is the target feature upgrade network with fixed parameters after training. The feature upgrade network is trained by using the first feature matrix extracted by the lightweight network model deployed on the end-side device, and the universal network model combined with the trained feature upgrade network is trained using the image set stored in the cloud, thereby avoiding the dependence of the universal network model on the images collected by the end-side device. At the same time, the feature extraction network is optimized based on offline (first feature matrix) and cloud data (cloud image set), so that it has a better supervision effect on the features output by the front-end network, which is conducive to the further convergence of the training results of complex scene instances.
[0053] Step S206, training the i-th lightweight network model by using the first image recognition result and the image set to obtain an i+1-th lightweight network model;
[0054] The above-mentioned first image recognition result can be the result of the output of the general network model. The lightweight network model is trained through an image set, and the output result of the training is compared with the first image recognition result to determine the lightweight network model and complete the upgrade and evolution of the lightweight network model. In this way, when the end-side device does not have the computing power for training, the evolution of the end-side device deployment model can be completed, thereby realizing end-cloud collaborative evolution.
[0055] Specifically, the i-th lightweight network model is trained for the jth round through the image set to obtain a second image recognition result output by the j-th round of training of the i-th lightweight network model, wherein j is an integer greater than or equal to 1; the similarity between the second image recognition result and the first image recognition result is calculated to obtain a calculation result; when the calculation result is greater than or equal to a preset similarity threshold, the lightweight network model obtained by the j-th round of training is determined as the i+1-th lightweight network model; when the calculation result is less than the preset similarity threshold, the i-th lightweight network model is trained for the j+1-th round.
[0056] When the value of i is 1, the i-th lightweight network model is the initial lightweight network model. The initial lightweight network model can be a network model of any structure with the same input, output and function as the general network model, or it can be a network model that has been processed by model compression and acceleration technology (such as pruning, quantization, knowledge distillation, etc.) on the general network model.
[0057] For example, Figure 7 is a training flow chart of a lightweight network model according to an embodiment of the present invention, such as Figure 7 As shown, the image is input into the i+1th general network model, and the first image recognition result (F1) output by the i+1th general network model can be obtained. At the same time, the image is sent to the i-th lightweight network model for learning and training, wherein the i-th lightweight network model will output a second image recognition result (F2) for each round of training. By calculating the similarity between F1 and F2, if the similarity is greater than or equal to the preset similarity threshold, it means that the training of the i-th lightweight network model is completed, and the training is stopped to obtain the i+1th lightweight network model; if the similarity is less than the preset similarity threshold, it means that the training of the i-th lightweight network model is not completed, and the next round of training is continued, and so on, until the training of the i-th lightweight network model is completed, and the evolution of the lightweight network model is realized. As a result, the lightweight network model deployed on the end-side device has a strong recognition ability for complex scenes, and the recognition accuracy and generalization ability of the lightweight network model are improved.
[0058] Step S208: Identify the target object in the image captured by the terminal side device through the (i+1)th lightweight network model.
[0059] Optionally, the executor of the above steps may be a background processor, or other devices with similar processing capabilities, or a machine that integrates at least an image acquisition device and a data processing device, wherein the image acquisition device may include a graphics acquisition module such as a camera, and the data processing device may include a computer, a mobile phone or other terminal, but is not limited thereto.
[0060] Through the above steps, the image set extracted by the lightweight network model deployed on the end device is used to train the feature upgrade network, and the cloud image set is used to train the feature extraction network, thereby completing the evolution of the general network model deployed on the cloud; at the same time, the evolution of the lightweight network model is completed according to the evolved general network model and the image set extracted by the lightweight network model, thereby realizing the co-evolution of the end-cloud model. This solves the problem of limited model recognition capability of the end-side device deployment in the related technology and improves the recognition capability of the model deployed on the end-side device.
[0061] As an optional implementation, according to the signal-to-noise ratio of the image feature vector, the image feature vectors in each of the image feature vector subsets are screened to obtain N target image feature vector subsets, including: performing the following operations on each image feature vector in the image feature vector subset, and determining the image feature vector performing the following operations as the current image feature vector: determining the signal-to-noise ratio of the current image feature vector; determining that the current image feature vector is valid when the signal-to-noise ratio is greater than or equal to a preset signal-to-noise ratio threshold; determining that the current image feature vector is invalid and deleting the current image feature vector when the signal-to-noise ratio is less than a preset signal-to-noise ratio threshold; and determining the valid image feature vector as the image feature vector in the target image feature vector subset.
[0062] The above-mentioned preset signal-to-noise ratio threshold can be set according to actual conditions, such as 3, 4, 5, etc. The larger the signal-to-noise ratio, the better the validity of the feature vector; the image feature vectors in the image feature vector subset are screened by the signal-to-noise ratio, and when the signal-to-noise ratio of the image feature vector is greater than or equal to the preset signal-to-noise ratio threshold, it indicates that the image feature vector is valid, and at this time, the image feature vector is retained; when the signal-to-noise ratio of the image feature vector is less than the preset signal-to-noise ratio threshold, it indicates that the image feature vector is invalid, and at this time, the image feature vector needs to be deleted, and the image feature vectors in the image feature vector subset are screened to finally obtain the target image feature vector subset, and the target image feature vector subset is determined as the first feature vector.
[0063] As an optional implementation, Figure 8 is a specific flow chart of the evolution of the end-cloud model system according to an embodiment of the present invention, such as Figure 8 As shown, the specific process is as follows:
[0064] Cloud:
[0065] S811, training the general network model and feature remapping module (including feature mapping network) deployed in the cloud based on the offline data distribution (first feature matrix) and cloud data (cloud image set) as the basic model to achieve the evolution of the general network model;
[0066] S812, based on model compression and acceleration technology, generates a certain number of lightweight network models based on the evolved general network model for deployment on the end-side devices, thus realizing the evolution of lightweight network models;
[0067] S813, complete the deployment of the evolved general network model, evolved lightweight network model, and feature remapping module;
[0068] S814, end-cloud model collaborative upgrade completed.
[0069] Client-side equipment:
[0070] S821: Obtain an image set through the terminal device, and extract features from the image set using the deployed lightweight network model to obtain an image feature vector set.
[0071] S822, performing clustering, merging and other operations on the image feature vectors to obtain a subset of the image feature vectors, and uploading the subset of the image feature vectors to the cloud;
[0072] S823, screening the image feature vector subset, judging the validity of each feature vector in the image feature vector subset, retaining the valid feature vectors, and determining the first feature matrix (i.e., the archive feature);
[0073] S824, performing end-cloud feature space mapping on the first feature matrix through a feature remapping module to obtain a third feature matrix;
[0074] S825, training (upgrading) the general network model deployed in the cloud based on the cloud data and the third feature matrix (the mapped first feature matrix), and completing the evolution of the general network model.
[0075] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.
[0076] In this embodiment, a device for recognizing an image is also provided, and the device is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware is also possible and conceived.
[0077] Fig. 9 is a structural block diagram of an apparatus for recognizing an image according to an embodiment of the present invention, such as Fig. 9As shown, the device includes: an acquisition module 902, which is used to input the image set collected by the terminal device into the i-th lightweight network model, and obtain the first feature matrix output by the i-th lightweight network model, wherein the i-th lightweight network model is a model obtained after the initial lightweight network model is trained for the i-th time, wherein i is an integer greater than or equal to 1, and the lightweight network model is deployed on the terminal device; a first training module 904, which is used to train the i-th general network model for i+1 times through the first feature matrix to obtain the first image recognition matrix output by the i+1-th general network model. Results, wherein the i-th general network model is a model obtained after the initial general network model is trained for the i-th time, the i+1-th general network model is a model obtained after the i-th general network model is trained for the i+1-th time, and the general network model is deployed in the cloud; the second training module 906 is used to train the i-th lightweight network model through the first image recognition result and the image set to obtain the i+1-th lightweight network model; the recognition module 908 is used to recognize the target object in the image collected by the terminal side device through the i+1-th lightweight network model.
[0078] In an exemplary embodiment, the device is also used to input the first feature matrix into a feature mapping network, and convert the first feature matrix into a third feature matrix through the feature mapping network, wherein the feature mapping network is located between the general network model and the lightweight network model; input the third feature matrix into the feature upgrade network of the i-th general network model, and train the feature upgrade network through the third feature matrix to obtain a target feature upgrade network.
[0079] In an exemplary embodiment, the device is also used to train the i-th feature extraction network and the target feature upgrading network using the cloud image set to obtain the i+1-th general network model, wherein the cloud image set is the image set stored in the cloud, the general network model includes a feature extraction network and a feature upgrading network, the output of the feature extraction network is the input of the feature upgrading network, and the i-th feature extraction network is the feature extraction network in the i-th general network model.
[0080] In an exemplary embodiment, the apparatus is further used to perform feature extraction on images in the image set through the i-th lightweight network model to obtain a set of image feature vectors; divide the set of image feature vectors into N image feature vector subsets, each of the image feature vector subsets including at least one image feature vector of the same sample object, the sample object is an object in the image collected by the end-side device, and N is an integer greater than or equal to 1; and obtain the first feature matrix through the N image feature vector subsets.
[0081] In an exemplary embodiment, the device is also used to screen the image feature vectors in each of the image feature vector subsets according to the signal-to-noise ratio of the image feature vector to obtain N target image feature vector subsets; and determine the N target image feature vector subsets as the first feature matrix.
[0082] In an exemplary embodiment, the device is also used to perform the following operations on each image feature vector in the image feature vector subset, and determine the image feature vector that performs the following operations as the current image feature vector: determine the signal-to-noise ratio of the current image feature vector; when the signal-to-noise ratio is greater than or equal to a preset signal-to-noise ratio threshold, determine that the current image feature vector is valid; when the signal-to-noise ratio is less than a preset signal-to-noise ratio threshold, determine that the current image feature vector is invalid and delete the current image feature vector; determine the valid image feature vector as the image feature vector in the target image feature vector subset.
[0083] In an exemplary embodiment, the device is also used to perform the jth round of training on the i-th lightweight network model through the image set to obtain a second image recognition result output by the j-th round of training of the i-th lightweight network model, wherein j is an integer greater than or equal to 1; calculate the similarity between the second image recognition result and the first image recognition result to obtain a calculation result; when the calculation result is greater than or equal to a preset similarity threshold, determine the lightweight network model obtained by the j-th round of training as the i+1-th lightweight network model; when the calculation result is less than the preset similarity threshold, perform the j+1-th round of training on the i-th lightweight network model.
[0084] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.
[0085] An embodiment of the present invention further provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above methods are implemented.
[0086] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.
[0087] An embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.
[0088] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.
[0089] An embodiment of the present invention further provides a computer program product, including a computer program, which implements the steps of the method described in each embodiment of the present application when executed by a processor.
[0090] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.
[0091] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.
[0092] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A method for recognizing an image, characterized in that: include: Inputting an image set collected by an end-side device into an i-th lightweight network model, and obtaining a first feature matrix output by the i-th lightweight network model, wherein the i-th lightweight network model is a model obtained after the i-th training of the initial lightweight network model, wherein i is an integer greater than or equal to 1, and the lightweight network model is deployed on the end-side device; The i-th general network model is trained i+1 times by using the first feature matrix to obtain a first image recognition result output by the i+1-th general network model, wherein the i-th general network model is a model obtained after the initial general network model is trained for the i-th time, the i+1-th general network model is a model obtained after the i-th general network model is trained for the i+1-th time, and the general network model is deployed in the cloud; Training the i-th lightweight network model by using the first image recognition result and the image set to obtain an i+1-th lightweight network model; Identifying a target object in the image captured by the terminal device through the i+1th lightweight network model; The i-th lightweight network model is trained by using the first image recognition result and the image set to obtain the i+1-th lightweight network model, including: Performing a j-th round of training on the i-th lightweight network model through the image set to obtain a second image recognition result output by the j-th round of training of the i-th lightweight network model, where j is an integer greater than or equal to 1; Calculating the similarity between the second image recognition result and the first image recognition result to obtain a calculation result; When the calculation result is greater than or equal to a preset similarity threshold, the lightweight network model obtained by the j-th round of training is determined as the i+1th lightweight network model; When the calculation result is less than the preset similarity threshold, the i-th lightweight network model is trained for the j+1th round.
2. The method according to claim 1, characterized in that: The i-th general network model is trained i+1 times by using the first feature matrix, including: Inputting the first feature matrix into a feature mapping network, and converting the first feature matrix into a third feature matrix through the feature mapping network, wherein the feature mapping network is located between the general network model and the lightweight network model; The third feature matrix is input into the feature upgrade network of the i-th general network model, and the feature upgrade network is trained by the third feature matrix to obtain a target feature upgrade network.
3. The method according to claim 2, characterized in that After obtaining the target feature upgrade network, the method further includes: The i-th feature extraction network and the target feature upgrading network are trained using a cloud image set to obtain the i+1-th general network model, wherein the cloud image set is a set of images stored in the cloud, the general network model includes a feature extraction network and a feature upgrading network, the output of the feature extraction network is the input of the feature upgrading network, and the i-th feature extraction network is the feature extraction network in the i-th general network model.
4. The method according to claim 1, characterized in that: Inputting an image set collected by a terminal device into an i-th lightweight network model, and obtaining a first feature matrix output by the i-th lightweight network model, including: Extracting features of images in the image set by using the i-th lightweight network model to obtain a set of image feature vectors; Dividing the image feature vector set into N image feature vector subsets, each of the image feature vector subsets includes at least one image feature vector of the same sample object, the sample object is an object in the image collected by the end-side device, and N is an integer greater than or equal to 1; The first feature matrix is obtained through the N image feature vector subsets.
5. The method according to claim 4, characterized in that Obtaining the first feature matrix through the N image feature vector subsets includes: Screening the image feature vectors in each of the image feature vector subsets according to the signal-to-noise ratio of the image feature vector to obtain N target image feature vector subsets; The N target image feature vector subsets are determined as the first feature matrix.
6. The method according to claim 5, characterized in that According to the signal-to-noise ratio of the image feature vector, the image feature vectors in each of the image feature vector subsets are screened to obtain N target image feature vector subsets, including: Perform the following operations on each image feature vector in the image feature vector subset, and determine the image feature vector that performs the following operations as the current image feature vector: Determining a signal-to-noise ratio of the current image feature vector; When the signal-to-noise ratio is greater than or equal to a preset signal-to-noise ratio threshold, determining that the current image feature vector is valid; When the signal-to-noise ratio is less than a preset signal-to-noise ratio threshold, determining that the current image feature vector is invalid, and deleting the current image feature vector; The valid image feature vector is determined as the image feature vector in the target image feature vector subset.
7. A device for recognizing an image, characterized in that: include: An acquisition module, used to input an image set collected by an end-side device into an i-th lightweight network model, and acquire a first feature matrix output by the i-th lightweight network model, wherein the i-th lightweight network model is a model obtained after the i-th training of the initial lightweight network model, wherein i is an integer greater than or equal to 1, and the lightweight network model is deployed on the end-side device; A first training module is used to train the i-th general network model for i+1 times through the first feature matrix to obtain a first image recognition result output by the i+1-th general network model, wherein the i-th general network model is a model obtained after the i-th training of the initial general network model, the i+1-th general network model is a model obtained after the i-th training of the i-th general network model, and the general network model is deployed in the cloud; A second training module is used to train the i-th lightweight network model through the first image recognition result and the image set to obtain an i+1-th lightweight network model; an identification module, configured to identify a target object in the image collected by the terminal device through the i+1th lightweight network model; The device is also used to perform the jth round of training on the i-th lightweight network model through the image set to obtain a second image recognition result output by the j-th round of training of the i-th lightweight network model, wherein j is an integer greater than or equal to 1; calculate the similarity between the second image recognition result and the first image recognition result to obtain a calculation result; when the calculation result is greater than or equal to a preset similarity threshold, determine the lightweight network model obtained by the j-th round of training as the i+1-th lightweight network model; when the calculation result is less than the preset similarity threshold, perform the j+1-th round of training on the i-th lightweight network model.
8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program implements the steps of the method described in any one of claims 1 to 6 when executed by a processor.
9. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to perform the method according to any one of claims 1 to 6.
