Model deployment method, apparatus, system, and electronic device

Abstract

A model deployment method, an apparatus, a system, and an electronic device, relating to the technical field of artificial intelligence. The method comprises: acquiring generation requirement information, inputted by a user, for a network model to be deployed in a first terminal; performing semantic analysis on the generation requirement information to obtain a semantic analysis result, the semantic analysis result comprising first model information characterizing an identification subject of the network model to be deployed; retrieving from a preset image library on the basis of the first model information to obtain a training image comprising the identification subject, the preset image library comprising: a first image library storing historical images uploaded by the first terminal, and / or a second image library storing historical images uploaded by a user-associated second terminal other than the first terminal; and on the basis of the training image, training a network model having an initial structure to obtain a target network model, and sending the target network model to the first terminal. In this way, the detection accuracy of the network model deployed on the terminal is improved.

Description

Model deployment method, device, system and electronic device

[0001] The present application claims priority to the Chinese patent application No. 202411337962.6, filed on September 24, 2024, and entitled "A model deployment method, device, system and electronic device", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application relates to the field of artificial intelligence, and in particular, to a model deployment method, device, system and electronic device. BACKGROUND

[0003] With the development of artificial intelligence technology, a terminal (for example, a household camera, a driving recorder, etc.) can perform corresponding tasks by deploying network models with different functions. For example, a driving recorder can deploy a vehicle detection model to detect vehicles appearing in the captured pictures.

[0004] However, the network models deployed in the terminal are usually pre-trained by the developers in the production stage. In the process of model training, the training data used is usually data in the public training set, so that the network model deployed in the terminal does not match the scene where the terminal is located, resulting in low accuracy of the detection result of the network model. SUMMARY

[0005] The purpose of the embodiments of the present application is to provide a model deployment method, device, system and electronic device to improve the accuracy of the detection of the network model deployed in the terminal. The specific technical solutions are as follows:

[0006] In a first aspect, the present application provides a model deployment method, which comprises:

[0007] obtaining the generation requirement information of the network model to be deployed in the first terminal input by the user;

[0008] performing semantic analysis on the generation requirement information to obtain a semantic analysis result; wherein the semantic analysis result contains first model information representing the recognition subject of the network model to be deployed;

[0009] performing retrieval in a preset image library based on the first model information to obtain training images containing the recognition subject; wherein the preset image library contains a first image library for storing historical images uploaded by the first terminal, and / or a second image library for storing historical images uploaded by a second terminal associated with the user and other than the first terminal;

[0010] training, based on the training image, a network model of an initial structure to obtain a target network model, and sending the target network model to the first terminal.

[0011] In an optional embodiment, the method is applied to a server in which a large language model is deployed.

[0012] The semantic analysis of the generated demand information obtains a semantic analysis result; the first model information is used to search a preset image library to obtain a training image containing the identified subject; and a network model of an initial structure is trained based on the training image to obtain a target network model, which is sent to the first terminal.

[0013] The generated demand information is input into the large language model to perform semantic analysis on the generated demand information by the large language model to obtain a semantic analysis result; the first model information is used to search a preset image library to obtain a training image containing the identified subject; and a network model of an initial structure is trained based on the training image to obtain a target network model, which is sent to the first terminal.

[0014] In an optional embodiment, the searching, based on the first model information, of the preset image library to obtain a training image containing the identified subject comprises:

[0015] An image in the first image library that matches the first model information is determined as a first candidate image.

[0016] If the number of the first candidate images is greater than a preset threshold, the first candidate images are determined as training images.

[0017] In an optional embodiment, the method further comprises:

[0018] If the number of the first candidate images is not greater than the preset threshold, an image in the second image library that matches the first model information is determined as a second candidate image.

[0019] If the sum of the number of the first candidate images and the second candidate images is greater than the preset threshold, the first candidate images and the second candidate images are determined as training images.

[0020] In an optional embodiment, the method further comprises:

[0021] If the sum of the number of the first candidate images and the second candidate images is not greater than the preset threshold, an image in a preset public image library that matches the first model information is determined as a third candidate image.

[0022] The first candidate image, the second candidate image, and the third candidate image are determined as training images.

[0023] In an optional embodiment, the images in the preset public image library are consistent with the scenes of the images in the first image library.

[0024] In an optional embodiment, for any image library, the images in the image library that match the first model information are determined by the following steps:

[0025] An image with a subject label is determined from the image library as a labeled image;

[0026] An image in which the subject label contains the identified subject is determined from the labeled image as a first matching image;

[0027] The first model information is feature-extracted based on a text encoder in a pre-trained image-text retrieval model to obtain a text feature of the first model information;

[0028] An unlabeled image in the image library whose image feature matches the text feature is determined as a second matching image; wherein the image feature of any unlabeled image in the image library is obtained by feature-extracting the unlabeled image based on an image encoder in the image-text retrieval model;

[0029] The images in the image library that match the first model information are obtained in combination with the first matching image and the second matching image.

[0030] In an optional embodiment, before the retrieval based on the first model information in the preset image library to obtain the training image containing the identified subject, the method further comprises:

[0031] A reference image for the network model to be deployed input by a user is obtained;

[0032] The step of determining the images in the image library that match the first model information further comprises:

[0033] The reference image is feature-extracted to obtain an image feature of the reference image;

[0034] An image in the image library whose image feature matches the image feature of the reference image is determined as a third matching image; wherein the images in the image library that match the first model information further include the third matching image.

[0035] In an optional embodiment, before the retrieval based on the first model information in the preset image library to obtain the training image containing the identified subject, the method further comprises:

[0036] determining whether a network model satisfying the semantic analysis result exists currently;

[0037] if a network model satisfying the semantic analysis result exists, sending the network model satisfying the semantic analysis result to the first terminal;

[0038] The searching in the preset image library based on the first model information to obtain the training image containing the identified subject includes:

[0039] if a network model satisfying the semantic analysis result does not exist, searching in the preset image library based on the first model information to obtain the training image containing the identified subject.

[0040] In an optional embodiment, the semantic analysis result further contains second model information representing a task type of the network model to be deployed;

[0041] Before training the network model with the initial structure based on the training image to obtain the target network model, the method further includes:

[0042] determining, according to terminal information of the first terminal, a network model supporting deployment in the first terminal and having a task type consistent with the task type represented by the second model information as the network model with the initial structure.

[0043] In an optional embodiment, the generation requirement information contains terminal information of the first terminal.

[0044] or,

[0045] The terminal information of the first terminal is queried from pre-recorded terminal information of each terminal.

[0046] In an optional embodiment, the scene of the image collected by the first terminal is consistent with the scene of the image collected by the second terminal.

[0047] A second aspect of the embodiments of the present application provides a model deployment system, the system including: a first terminal and a server;

[0048] The first terminal is configured to acquire generation requirement information of a network model to be deployed in the first terminal input by a user, and send the generation requirement information to the server.

[0049] The server is configured to execute the model deployment method of any one of the first aspect.

[0050] The first terminal is further configured to receive the network model from the server.

[0051] In a third aspect, the embodiment of the present application provides a model deployment device, the device comprising:

[0052] A generation requirement information acquisition module is configured to acquire generation requirement information of a network model to be deployed in a first terminal input by a user;

[0053] A semantic analysis module is configured to perform semantic analysis on the generation requirement information to obtain a semantic analysis result, wherein the semantic analysis result comprises first model information representing an identified subject of the network model to be deployed;

[0054] A first retrieval module is configured to perform retrieval in a preset image library based on the first model information to obtain training images containing the identified subject, wherein the preset image library comprises a first image library configured to store historical images uploaded by the first terminal and / or a second image library configured to store historical images uploaded by a second terminal associated with the user and different from the first terminal;

[0055] A training module is configured to train an initial structure network model based on the training images to obtain a target network model and send the target network model to the first terminal.

[0056] In an optional embodiment, the method is applied to a server, and the server is deployed with a large language model;

[0057] The semantic analysis module, the first retrieval module, and the training module are specifically configured to:

[0058] input the generation requirement information into the large language model to perform semantic analysis on the generation requirement information by the large language model to obtain a semantic analysis result, perform retrieval in a preset image library based on the first model information to obtain training images containing the identified subject, and train an initial structure network model based on the training images to obtain a target network model and send the target network model to the first terminal.

[0059] In an optional embodiment, the first retrieval module comprises:

[0060] determine images in the first image library that match the first model information as first candidate images;

[0061] if the number of the first candidate images is greater than a preset threshold, determine the first candidate images as training images.

[0062] In an optional embodiment, the device further comprises:

[0063] a second candidate image determination module configured to, if the number of the first candidate images is not greater than the preset threshold, determine images in the second image library matching the first model information as second candidate images;

[0064] a first training image determination module configured to, if the sum of the number of the first candidate images and the number of the second candidate images is greater than the preset threshold, determine the first candidate images and the second candidate images as training images.

[0065] In an optional embodiment, the apparatus further comprises:

[0066] a third candidate image determination module configured to, if the sum of the number of the first candidate images and the number of the second candidate images is not greater than the preset threshold, determine images in a preset common image library matching the first model information as third candidate images;

[0067] a second training image determination module configured to determine the first candidate images, the second candidate images, and the third candidate images as training images.

[0068] In an optional embodiment, the images in the preset common image library are consistent with the scenes of the images in the first image library.

[0069] In an optional embodiment, for any image library, the images in the image library matching the first model information are determined by the following steps:

[0070] determining images with subject labels from the image library as labeled images;

[0071] determining, from the labeled images, images in which the subject labels contain the identified subject as first matching images;

[0072] performing feature extraction on the first model information based on a text encoder in a pre-trained image-text retrieval model to obtain text features of the first model information;

[0073] determining unlabeled images in the image library in which image features match the text features as second matching images; wherein the image features of any unlabeled image in the image library are obtained by performing feature extraction on the unlabeled image based on an image encoder in the image-text retrieval model;

[0074] combining the first matching images and the second matching images to obtain the images in the image library matching the first model information.

[0075] In an optional embodiment, before the searching in the preset image library based on the first model information to obtain the training image containing the recognition subject, the method further comprises:

[0076] obtaining a reference image input by a user for the network model to be deployed;

[0077] The step of determining the image in the image library matching the first model information further comprises:

[0078] performing feature extraction on the reference image to obtain image features of the reference image;

[0079] determining an image in the image library whose image features match the image features of the reference image as a third matching image; wherein the image in the image library matching the first model information further comprises the third matching image.

[0080] In an optional embodiment, the apparatus further comprises:

[0081] a first determining module configured to determine whether there is a network model satisfying the semantic analysis result before the searching in the preset image library based on the first model information to obtain the training image containing the recognition subject;

[0082] a first issuing module configured to send the network model satisfying the semantic analysis result to the first terminal if there is a network model satisfying the semantic analysis result;

[0083] The first searching module is specifically configured to:

[0084] if there is no network model satisfying the semantic analysis result, search in the preset image library based on the first model information to obtain the training image containing the recognition subject.

[0085] In an optional embodiment, the semantic analysis result further comprises second model information representing a task type of the network model to be deployed;

[0086] The apparatus further comprises:

[0087] an initial network model determining module configured to, before training the network model of initial structure based on the training image to obtain the target network model, determine a network model supporting deployment on the first terminal and having a task type consistent with the task type represented by the second model information as the network model of initial structure according to terminal information of the first terminal.

[0088] In an optional embodiment, the generation requirement information comprises terminal information of the first terminal; or the terminal information of the first terminal is queried from pre-recorded terminal information of each terminal.

[0089] In an optional embodiment, the first terminal and the second terminal collect images of the same scene.

[0090] In a fourth aspect, an electronic device is provided, comprising:

[0091] a memory for storing a computer program;

[0092] a processor for executing the program stored in the memory, thereby implementing the model deployment method described above.

[0093] In another aspect, a non-transitory computer readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the model deployment method described above is implemented.

[0094] In another aspect, a computer program product comprising instructions which, when executed on a computer, cause the computer to carry out the model deployment method described above.

[0095] The embodiments of the present application have the following beneficial effects:

[0096] The embodiments of the present application provide a model deployment method. A user can input generation requirement information indicating deployment of a network model on a first terminal according to actual requirements. Then, an electronic device can perform semantic analysis on the obtained generation requirement information to obtain a semantic analysis result. The semantic analysis result comprises first model information representing an identified subject of the network model to be deployed. Therefore, the first model information can be used to search a preset image library to obtain training images containing the identified subject. Since the images in the preset image library (i.e., the first image library and / or the second image library) are all historical images uploaded by terminals associated with the user, the images in the preset image library conform to the actual use scenario of the network model to be deployed. Accordingly, the determined training images are closer to the images actually collected by the first terminal. Then, the initial structure network model is trained based on the determined training images to obtain a target network model. The target network model can meet the user's requirements in terms of the identified subject and the use scenario, and can ensure that the target network model deployed on the first terminal can accurately process the images collected by the first terminal according to the user's requirements, i.e., improve the accuracy of the network model detection deployed on the terminal.

[0097] Of course, implementing any product or method of the present application does not necessarily require all the advantages described above to be achieved at the same time. Attached Figure Description

[0098] The accompanying drawings, which are provided to further illustrate this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application.

[0099] Figure 1 is a schematic diagram of the model deployment system provided in this application;

[0100] Figure 2 is a flowchart of the first type of model deployment method provided in the embodiments of this application;

[0101] Figure 3 is a schematic diagram of a model deployment system provided in this application;

[0102] Figure 4 is a schematic diagram of another model deployment system provided in this application;

[0103] Figure 5 is a flowchart of the present application for determining an image in an image library that matches the information of the first model.

[0104] Figure 6 is a second flowchart of the model deployment method provided in the embodiments of this application;

[0105] Figure 7 is a flowchart of a model deployment provided in an embodiment of this application;

[0106] Figure 8 is a structural diagram of a model deployment device provided in an embodiment of this application;

[0107] Figure 9 is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0108] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided with reference to the accompanying drawings and embodiments. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments in this application are within the scope of protection of this application.

[0109] With the development of artificial intelligence technology, terminals (such as home cameras and dashcams) can perform corresponding tasks by deploying network models with different functions. For example, dashcams can deploy vehicle detection models to detect vehicles appearing in the captured footage.

[0110] However, most network models deployed on terminals are pre-trained by developers during the production phase. During model training, the training data used is often data from public training sets, making it difficult for the network models deployed on terminals to meet users' personalized needs.

[0111] The embodiment of the present application provides a model deployment method, which can be applied to an electronic device, and the electronic device can be a server in a model deployment system. Referring to FIG. 1, FIG. 1 is a schematic diagram of a model deployment system provided by the present application. A model deployment system 100 comprises a server 101 and at least one terminal 102. The terminal 102 can be a household camera, a dashboard recorder or other intelligent devices capable of collecting images. The server 101 and each terminal 102 can transmit data. The server 101 can obtain generation requirement information input by a user for a specified terminal (i.e. a first terminal), and send a network model meeting the user requirement to the first terminal according to the model deployment method provided by the present application, so that the first terminal deploys the network model. The first terminal can be any terminal 102 in the system 100.

[0112] Referring to FIG. 2, FIG. 2 is a first flowchart of the model deployment method provided by the embodiment of the present application, and the method comprises the following steps:

[0113] S201: Obtain generation requirement information of a network model to be deployed in a first terminal input by a user.

[0114] S202: Perform semantic analysis on the generation requirement information to obtain a semantic analysis result.

[0115] The semantic analysis result comprises first model information representing an identified subject of the network model to be deployed.

[0116] S203: Perform retrieval in a preset image library based on the first model information to obtain training images containing the identified subject.

[0117] The preset image library comprises a first image library for storing historical images uploaded by the first terminal, and / or a second image library for storing historical images uploaded by a second terminal associated with the user and different from the first terminal.

[0118] S204: Train an initial structure network model based on the training images to obtain a target network model, and send the target network model to the first terminal.

[0119] The embodiment of the application provides a model deployment method, a user can input generation requirement information indicating deployment of a network model in a first terminal according to actual requirements, and then an electronic device can perform semantic analysis on the obtained generation requirement information to obtain a semantic analysis result. The semantic analysis result contains first model information representing an identification subject of the network model to be deployed, so that the first model information can be used for searching in a preset image library to obtain training images containing the identification subject. Since the images in the preset image library (i.e., the first image library and / or the second image library) are all historical images uploaded by terminals associated with the user, the images in the preset image library meet the actual use scenario of the network model to be deployed, and accordingly, the determined training images are closer to the images actually collected by the first terminal. Then, the initial structure network model is trained based on the determined training images, and the obtained target network model can meet the user requirements in terms of the identification subject and the use scenario, so as to ensure that the target network model deployed in the first terminal can accurately process the images collected by the first terminal according to the user requirements, that is, the accuracy of the network model detection deployed in the terminal is improved.

[0120] For step S201, the first terminal represents the terminal indicated by the user for the current required network model.

[0121] It can be understood that one user can be associated with multiple terminals, wherein the terminals associated with the user represent terminals controlled by the user. For example, a user can install multiple terminals (which can be cameras) in different areas of a home, and the multiple terminals are the terminals associated with the user. Accordingly, the user can control the multiple terminals through a software application installed on a smart device to browse the image pictures collected by the multiple terminals. For example, the smart device can be an electronic device such as a desktop computer, a notebook computer, a tablet computer, a smart phone, etc.

[0122] Accordingly, the user can select one terminal from all the associated terminals as the terminal (i.e., the first terminal) for the current required network model according to actual requirements. Then, the user can input generation requirement information for the first terminal to deploy the required network model in the first terminal.

[0123] The generation requirement information at least contains information describing the model function of the network model required by the user. For example, the generation requirement information can be: "deploy a neural network model capable of identifying squirrels", and for another example, the generation requirement information can be: "enable the device to support squirrel detection".

[0124] In addition, since one user can be associated with multiple terminals, generating the requirement information can further include terminal information describing a terminal to be deployed in the network model, which is used to determine the first terminal from the multiple terminals associated with the user. For example, the terminal information of one terminal can include an ID (Identity Document) of the terminal, a model of the terminal, a user-defined name corresponding to the terminal, and a user ID associated with the terminal, etc. In actual application scenarios, the terminal information of one terminal can include at least one of the above information.

[0125] For example, in the above scenario where one user is associated with multiple cameras, one camera is installed in a garden area of the user's home, and one camera is installed in a door area of the user's home. The user can pre-set the user-defined name of the camera installed in the garden area as camera 1 and the user-defined name of the camera installed in the door area as camera 2 through the software application installed on the smart device. Accordingly, the user input requirement information can be: "deploy a neural network model capable of recognizing squirrels in camera 1".

[0126] In this application, the server can also communicate with the software application installed on the smart device. Accordingly, for each user, the server can record and update the terminal information of each terminal associated with the user. That is, for each user, the user can send the terminal information of a terminal to the server through the software application installed on the smart device when the terminal is first associated, so that the server records the terminal information of each terminal associated with the user.

[0127] Subsequently, when the terminal information of one terminal associated with the user changes, the user can send the updated terminal information of the terminal to the server through the software application installed on the smart device, so that the server updates the recorded terminal information of the terminal associated with the user. For example, the user can modify the user-defined name of the terminal through the software application installed on the smart device. Accordingly, the software application can send the updated user-defined name of the terminal to the server, so that the server updates the recorded user-defined name of the terminal associated with the user.

[0128] The modalities of generating the requirement information include voice and / or text. In the case where the user input requirement information is voice, the electronic device (such as the server) can first perform voice recognition on the voice to convert the voice modality of the requirement information into the text modality of the requirement information. Then, the text modality of the requirement information is processed according to the above step S202.

[0129] Correspondingly, the user can input the generation requirement information in at least one of the following manners:

[0130] Manner 1: The user can directly input the generation requirement information through a terminal (i.e., the first terminal) required to deploy a network model. For example, the first terminal can be installed with a sound receiving device (such as a microphone) for receiving user voice and / or a display device (such as a display screen) for obtaining user input text.

[0131] Referring to FIG. 3, FIG. 3 is a schematic diagram of a scene of a model deployment system provided by the present application. In FIG. 3, terminal 1 is a camera installed with a sound receiving device (microphone), and the user can directly say to terminal 1 after turning on the sound receiving device of terminal 1: “I want the device to support the function of detecting cats”. Correspondingly, terminal 1 can send the voice modality generation requirement information input by the user to the server, realizing information uploading. Further, after obtaining the generation requirement information of the network model to be deployed in the first terminal (i.e., terminal 1) input by the user, the server can train a target network model according to the model deployment method provided by the present application, and then send the network model (i.e., the target network model) meeting the user’s requirement to terminal 1, realizing model distribution.

[0132] Manner 2: The user can input the generation requirement information for the first terminal through a software application installed on other smart devices.

[0133] Referring to FIG. 4, FIG. 4 is a schematic diagram of another scene of a model deployment system provided by the present application. In FIG. 4, terminal 2, terminal 3 and terminal 4 are all terminals associated with the user. Among them, terminal 2 and terminal 3 are cameras installed in the garden area of the user’s home, and terminal 4 is a camera installed in the entrance area of the user’s home. The user can input the generation requirement information “terminal 2 detects squirrels, and terminal 4 detects people” through a smart device installed with a software application. At this time, terminal 2 and terminal 4 can be used as the first terminal respectively. Correspondingly, after obtaining the generation requirement information input by the user, the server can train a target network model for terminal 2 and a target network model for terminal 4 according to the model deployment method provided by the present application, and then send the network model meeting the user’s requirement to terminal 2 and terminal 4 respectively, i.e., send the target network model for terminal 2 trained to terminal 2, and send the target network model for terminal 4 trained to terminal 4, realizing model distribution.

[0134] In some embodiments, the generation requirement information contains terminal information of the first terminal; or the terminal information of the first terminal is queried from pre-recorded terminal information of each terminal.

[0135] As described above, the generation requirement information input by the user can contain the terminal information of the first terminal.

[0136] For example, in the scenario shown in FIG. 3, the user inputted generation requirement information can be: "I want the device to support the function of detecting cats, and the ID of the device is: 1122XXXX". That is, the user inputted generation requirement information contains the terminal information of terminal 1.

[0137] Alternatively, the server can record the terminal information of all terminals associated with each user in advance. Then, after receiving the user inputted generation requirement information, the server can query the terminal information of the first terminal in the terminal information of each terminal associated with the user.

[0138] For example, in the scenario shown in FIG. 4, the server can record the terminal information of terminal 2, the terminal information of terminal 3, and the terminal information of terminal 4 associated with the user. If the user inputted generation requirement information is "terminal 2 detects squirrels", the server can retrieve the terminal information of terminal 2, i.e., the terminal information of the first terminal, from the pre-recorded terminal information of each terminal associated with the user after obtaining the generation requirement information. That is, the server can determine the terminal corresponding to the user-defined name "terminal 2" as the first terminal.

[0139] In some embodiments, a large language model (LLM) is deployed in the server. Accordingly, the server can input the obtained generation requirement information into the large language model to perform steps S202-S204 through the large language model.

[0140] In the embodiments of the present application, the large language model deployed in the server can be referred to as a network model generation agent. The network model generation agent can be obtained by model training of a large language model based on an initial structure using self-supervised learning, semi-supervised learning, or transfer learning.

[0141] In this way, the training process of the network model can be completed through the large language model, realizing an automated model training process, without the need for technical personnel to directly perform model training and model evaluation, so that the neural network model can be generated more conveniently and efficiently according to the user's model generation requirement information.

[0142] For step S202, the electronic device (e.g., the server) can perform semantic analysis on the obtained generation requirement information to obtain a semantic analysis result containing first model information. The first model information represents the recognition subject of the network model to be deployed (i.e., the object type to be recognized by the network model).

[0143] It can be understood that the large language model can determine the recognition subject of the network model to be deployed according to the information describing the model function of the network model in the generation requirement information. For example, when the generation requirement information is "I want the device to support the function of detecting cats", the semantic analysis result obtained by performing semantic analysis on the generation requirement information is that the recognition subject of the network model to be deployed is represented by the first model information: cat.

[0144] In some embodiments, the semantic analysis result further includes second model information representing the task type of the network model to be deployed.

[0145] In the embodiments of the present application, the large language model can determine the task type of the network model to be deployed according to the information describing the model function of the network model in the generation requirement information.

[0146] For example, when the generation requirement information is "I want the device to support the function of detecting cats", the semantic analysis result obtained by performing semantic analysis on the generation requirement information is that the task type of the network model to be deployed is represented by the second model information: detection.

[0147] For another example, when the generation requirement information is "I want the device to be able to determine whether the image contains a cat", the semantic analysis result obtained by performing semantic analysis on the generation requirement information is that the task type of the network model to be deployed is represented by the second model information: classification.

[0148] For steps S203 to S204, for each terminal (for example, a camera), the terminal can upload the collected image to the server, and the server can record the image uploaded by the terminal as the historical image uploaded by the terminal.

[0149] For the first terminal, the images stored in the first image library are the historical images uploaded by the first terminal. The images stored in the second image library are the historical images uploaded by the second terminal. The second terminal is other terminal associated with the user to which the first terminal belongs, except the first terminal.

[0150] For example, in the scenario shown in the above Fig. 4, the server can record the historical images uploaded by terminal 2, the historical images uploaded by terminal 3, and the historical images uploaded by terminal 4. If the generation requirement information input by the user is "terminal 2 detects squirrels", at this time, the first terminal is terminal 2, the second terminal includes terminal 3 and terminal 4, the images stored in the first image library represent the historical images uploaded by terminal 2, and the images stored in the second image library represent the historical images uploaded by terminal 3 and the historical images uploaded by terminal 4.

[0151] Correspondingly, after obtaining the first model information representing the identification subject of the network model to be deployed, the electronic device can perform retrieval in the preset image library (i.e., the first image library and / or the second image library) to obtain the training image containing the identification subject.

[0152] Further, the electronic device can implement training of the target network model based on the retrieved training image. That is, the initial-structure network model is trained based on the training image to obtain the target network model, and the target network model is sent to the first terminal. For example, after training the target network model, the electronic device can send the model file of the target network model to the first terminal, and the model file can represent the model architecture and model parameters of the target network model. Correspondingly, after receiving the model file, the first terminal can locally deploy the target network model.

[0153] In an implementation manner, before step S204, the method further includes:

[0154] According to the terminal information of the first terminal, a network model supporting deployment in the first terminal and having a task type consistent with the task type represented by the second model information is determined as the initial-structure network model.

[0155] It can be understood that, due to the constraint of the hardware condition of the terminal, the network model supported by the first terminal for deployment can be limited. For example, the first terminal can only support deployment of a network model whose required storage space is less than a preset storage space threshold. For another example, the first terminal can only support deployment of a network model of a certain fixed network structure.

[0156] Therefore, according to the terminal information of the first terminal and the second model information representing the task type, the server can determine a network model supporting deployment and having a task type consistent with the task type represented by the second model information as the initial-structure network model. For example, according to the second model information representing the task type, the server can determine a candidate network model having a task type consistent with the task type represented by the second model information, and then, according to the terminal information of the first terminal, determine a network model supporting deployment in the first terminal from the determined candidate network model as the initial-structure network model.

[0157] For example, if the second model information representing the task type is detection, the server can determine YOLOv5 as the initial-structure network model according to the model number in the terminal information of the first terminal. For example, if the second model information representing the task type is detection, the candidate network model determined by the server can include YOLOv5 and YOLOX, and then, according to the model number in the terminal information of the first terminal, the server can determine that the first terminal supports YOLOv5, and the server can determine YOLOv5 as the initial-structure network model.

[0158] If the second model information of the task type to be characterized is classification, the server can determine ResNet50 as the network model of the initial structure according to the model of the terminal information of the first terminal.

[0159] Based on the above processing, the initial structure of the network model used for training can be determined in combination with the terminal information of the first terminal and the second model information of the task type to be characterized, so that the target network model obtained by training can be deployed on the first terminal, so that the first terminal can be compatible with the target network model obtained by training, and the user can more conveniently deploy the target network model.

[0160] In an implementation manner, the electronic device can also determine corresponding training task parameters in combination with the terminal information of the first terminal and the second model information of the task type to be characterized. For example, the training task parameters include a learning rate, a sampling strategy, and a training paradigm used for model training, etc. For example, the sampling strategy can include random sampling or balanced sampling, etc. The training paradigm can include model distillation training, self-supervised learning, semi-supervised learning, and transfer learning, etc., and the learning rate can be 0.001.

[0161] In some embodiments, step S203 comprises:

[0162] Step 1: Determine the images in the first image library that match the first model information as first candidate images.

[0163] Step 2: If the number of first candidate images is greater than a preset threshold, the first candidate images are determined as training images.

[0164] In the embodiments of the present application, the retrieval of training images in the first image library can be given priority, that is, the images in the first image library that match the first model information are determined as first candidate images.

[0165] For any image library, the process of determining the images in the image library that match the first model information will be described in subsequent embodiments.

[0166] If the number of first candidate images is greater than a preset threshold, it indicates that the determined first candidate images are sufficient for the training of the network model, and therefore the first candidate images can be directly determined as training images. Wherein, the first candidate images being sufficient for the training of the network model means that the model parameters of the network model can achieve convergence in the iterative optimization process when the initial structure of the network model is trained based on the first candidate images.

[0167] Based on the above processing, since the images in the first image library are images collected by the first terminal itself, the first candidate images determined in the first image library are consistent with the position and angle of the images required to be processed when the network model to be deployed is actually applied. Correspondingly, using the determined first candidate images as training images to train the network model can improve the adaptability of the trained network model (i.e., the target network model) to the first terminal.

[0168] In some embodiments, the method further comprises:

[0169] Step 3: If the number of the first candidate images is not greater than the preset threshold, determining images in the second image library that match the first model information as second candidate images.

[0170] Step 4: If the sum of the number of the first candidate images and the number of the second candidate images is greater than the preset threshold, determining the first candidate images and the second candidate images as training images.

[0171] In the embodiments of the present application, if the number of the first candidate images is not greater than the preset threshold, it indicates that the number of the determined first candidate images is small and insufficient for training of the network model.

[0172] Further, since the images stored in the second image library represent historical images uploaded by the second terminal associated with the user to which the first terminal belongs in addition to the first terminal, it can be understood that the historical images uploaded by the second terminal are usually more consistent with the actual needs of the user since the second terminal and the first terminal both belong to the terminals associated with the user. Therefore, the retrieval of training images can be performed in the second image library, that is, determining images in the second image library that match the first model information as second candidate images.

[0173] In an implementation manner, the images collected by the first terminal and the second terminal are consistent in scene.

[0174] In the embodiments of the present application, the server can pre-determine a plurality of terminals that collect images consistent in scene among all the terminals associated with the same user. Further, the other terminals that collect images consistent in scene with the images collected by the first terminal are determined as the second terminal. For example, the terminal information of a terminal can further include the installation position of the terminal. The user can select the installation positions (such as a bedroom, a garden) of the terminals associated with the user through a software application installed on a smart device. Correspondingly, the server can determine a plurality of terminals with the same installation position as the installation positions of the terminals associated with the user recorded as the plurality of terminals that collect images consistent in scene.

[0175] For example, in the scenario shown in FIG. 4, since the terminal 2 and the terminal 3 are both installed in the garden area of the user's home, i.e., it can be considered that the scenes of the images collected by the terminal 2 and the terminal 3 are consistent. If the terminal 2 is the first terminal, the terminal 3 can be considered as the second terminal.

[0176] In this way, when sufficient training images cannot be determined based on the first image library alone, the training images can be further determined from the historical images collected by the second terminal associated with the same user and having consistent scenes, so as to supplement the training images determined from the first image library. In this way, the determined training images are close to the images to be processed when the network model to be deployed is actually applied, and the adaptability of the trained network model (i.e., the target network model) to the first terminal can be improved. For example, in the scenario shown in FIG. 4, when the images of the squirrel collected by the camera deployed in the garden, i.e., the terminal 2 as the first terminal, are insufficient, the images of the squirrel collected by the other camera deployed in the garden, i.e., the terminal 3 as the second terminal, can be used as the training images to train the network model for recognizing the squirrel. Since the terminal 3 is deployed in the same scene as the terminal 2, and the images collected by the terminal 3 are highly similar to the images collected by the terminal 2, the images collected by the terminal 3 can be better used as the supplement of the training images.

[0177] If the sum of the number of the first candidate images and the number of the second candidate images is greater than the preset threshold, it indicates that the determined first candidate images and the second candidate images are sufficient for training the network model, and thus the first candidate images and the second candidate images can be determined as the training images.

[0178] Based on the above processing, when sufficient training images cannot be determined based on the first image library alone, the training images can be further determined from the historical images collected by other terminals associated with the same user, so as to supplement the training images determined from the first image library. In this way, the determined training images are close to the images to be processed when the network model to be deployed is actually applied, and the adaptability of the trained network model (i.e., the target network model) to the first terminal can be improved.

[0179] In some embodiments, the method further includes:

[0180] Step 5: If the sum of the number of the first candidate images and the number of the second candidate images is not greater than the preset threshold, the images in the preset common image library that match the first model information are determined as the third candidate images.

[0181] Step 6: The first candidate images, the second candidate images, and the third candidate images are determined as the training images.

[0182] In the embodiments of the present application, if the sum of the number of the first candidate images and the number of the second candidate images is not greater than the preset threshold, it indicates that the sum of the number of the determined first candidate images and the number of the second candidate images is less, which is insufficient for the training of the network model.

[0183] At this time, in order to ensure that the number of the training images used for training the network model is sufficient, the retrieval of the training images in the preset public image library can be performed, that is, the images in the preset public image library that match the first model information are determined as the third candidate images.

[0184] In an implementation manner, the images in the preset public image library are consistent with the scenes of the images in the first image library.

[0185] In the embodiments of the present application, the server can determine the public image library that is consistent with the scenes of the images in the first image library. For example, in the scenario shown in FIG. 4, since the terminal 2 and the terminal 3 are both installed in the garden area of the user's home, if the terminal 2 is the first terminal, the scenes of the images in the preset public image library can be images of other gardens collected by other terminals except the terminal associated with the user.

[0186] In this way, when the historical images uploaded by the terminal associated with the user cannot determine sufficient training images, the training images can be further determined in the preset public image library that is consistent with the scenes of the collected images, to supplement the training images determined in the first image library and the second image library. The determined training images are close to the images required for processing when the network model to be deployed is actually applied, and the adaptability of the trained network model (that is, the target network model) to the first terminal can be improved.

[0187] Further, the first candidate images, the second candidate images, and the third candidate images can be determined as the training images.

[0188] Based on the above processing, when the historical images uploaded by the terminal associated with the user cannot determine sufficient training images, the training images can be further determined in the preset public image library, to supplement the training images determined in the first image library and the second image library, and to ensure the sufficiency of the training images. The trained network model (that is, the target network model) can realize the functions required by the user and meet the personalized needs of the user.

[0189] In some embodiments, referring to FIG. 5, FIG. 5 is a flowchart of determining images matching the first model information in an image library provided by the present application. Correspondingly, for any image library, the images matching the first model information in the image library are determined through the following steps:

[0190] S501: Determine the images with the subject label from the image library as the labeled images.

[0191] S502: Determine, from the labeled images, a subject label containing an image that identifies a subject as a first matching image.

[0192] S503: Perform feature extraction on the first model information based on a text encoder in a pre-trained image-text retrieval model to obtain text features of the first model information.

[0193] S504: Determine an unlabeled image in the image library that matches the text features as a second matching image.

[0194] Wherein the image features of any unlabeled image in the image library are obtained by performing feature extraction on the unlabeled image based on an image encoder in the image-text retrieval model.

[0195] S505: Combine the first matching image and the second matching image to obtain an image in the image library that matches the first model information.

[0196] In the embodiments of the present application, the image library can be any one of the above-mentioned first image library, second image library, and pre-set public image library. Correspondingly, for each image library, the technical personnel can pre-add a subject label to a part of the images in the image library. Correspondingly, the images with the subject label are labeled images, and the images without the subject label are unlabeled images.

[0197] For each labeled image, if the subject label of the labeled image contains the subject identified by the first model information, the labeled image can be determined as the first matching image.

[0198] It can be understood that for the unlabeled images, it is not possible to directly determine whether each unlabeled image matches the first model information by the above step S502. Therefore, based on the cross-modal image-text retrieval method, it can be determined whether each unlabeled image in the image library matches the first model information.

[0199] Correspondingly, the text features of the first model information can be obtained by performing feature extraction on the first model information based on a text encoder in a pre-trained image-text retrieval model. Wherein the image-text retrieval model is trained based on sample images and subject labels of each sample image, and the image-text retrieval model further includes an image encoder.

[0200] For each unlabeled image in the image library, the feature of the unlabeled image can be extracted in advance based on the image encoder in the image-text retrieval model. For example, the electronic device can extract the feature of the image after receiving the historical image uploaded by any terminal. Accordingly, during the matching process, the images can be matched directly according to the image features. In this way, the efficiency of matching can be improved.

[0201] Further, the unlabeled image in the image library whose image feature matches the text feature can be determined as the second matching image. For example, for each unlabeled image in the image library, the similarity between the image feature of the unlabeled image and the text feature of the first model information can be calculated, and the unlabeled image corresponding to the similarity greater than the preset similarity threshold can be determined as the second matching image.

[0202] In an implementation manner, after the first matching image and the second matching image in the image library are determined, the first matching image and the second matching image determined in the image library can be directly determined as the images in the image library matching the first model information.

[0203] Based on the above processing, for the labeled image, the image in the image library matching the first model information can be directly determined according to the subject label, and thus the efficiency of matching can be improved. For the unlabeled image, the image in the image library matching the first model information can be determined based on the text feature and the image feature through the cross-modal image-text retrieval manner, and thus the cross-modal matching can be realized, and the number of images matching the first model information determined can be increased.

[0204] In some embodiments, before step S203, the method further includes:

[0205] Obtaining the reference image input by the user for the network model to be deployed.

[0206] The above step of determining the image in the image library matching the first model information further includes:

[0207] Step a: extracting the feature of the reference image to obtain the image feature of the reference image.

[0208] Step b: determining the image in the image library whose image feature matches the image feature of the reference image as the third matching image.

[0209] The image in the image library matching the first model information further includes the third matching image.

[0210] In the embodiments of the present application, the user can also input a reference image for the network model to be deployed. In the process of determining the images in any image library that match the first model information, the reference image can also be combined to determine the images in the image library that match the first model information. The image library can be any one of the first image library, the second image library, and a preset public image library.

[0211] For example, when the user inputs the generation requirement information "deploy a neural network model capable of identifying squirrels in camera 1", the user can also input an image containing a squirrel in an image frame as a reference image.

[0212] Correspondingly, the reference image obtained can be subjected to feature extraction to obtain the image features of the reference image. The process of feature extraction on the reference image can refer to the process of feature extraction on the unlabeled image in the above embodiments, which will not be described here.

[0213] Further, the images in the image library whose image features match the image features of the reference image can be determined as third matching images. For example, for each image in the image library, the similarity between the image features of the image and the image features of the reference image can be calculated, and the image corresponding to the similarity greater than a preset similarity threshold can be determined as a third matching image.

[0214] Based on the above processing, in the case where the user inputs a reference image for the network model to be deployed, the reference image can be combined to determine the images in the image library that match the first model information. In this way, the number of images determined to match the first model information can be further improved. Moreover, matching according to the reference image input by the user can ensure that the images obtained by matching are more in line with the actual needs of the user, thereby improving the accuracy of the third matching images determined.

[0215] In some embodiments, if the user inputs a reference image, the reference image can also be used to search in the image library, and the searched images can be used as training images to train the network model of the initial structure, i.e., without combining the first matching images and the second matching images to determine the training images.

[0216] In other embodiments, if the user inputs a reference image, after the first matching images and the second matching images are determined, the images that match the reference image can be determined from the first matching images and the second matching images, and the determined images can be used as training images to train the network model of the initial structure.

[0217] In some embodiments, FIG. 6 is a second flowchart of a model deployment method provided by the embodiments of the present application. Based on FIG. 2, before step S203, the method further includes:

[0218] S205: determining whether there is a network model meeting the semantic analysis result.

[0219] If yes, there is a network model meeting the semantic analysis result, step S206 is performed; if no, there is no network model meeting the semantic analysis result, steps S203 to S204 are performed.

[0220] S206: sending the network model meeting the semantic analysis result to the first terminal.

[0221] Step S203 includes:

[0222] searching in the preset image library based on the first model information to obtain a training image containing the recognition subject.

[0223] It can be understood that the server can store a network model trained for other terminals in the past or a network model trained by a technician in advance.

[0224] In the embodiments of the present application, before searching for the training image according to step S203 described above, the server can determine whether there is a network model meeting the semantic analysis result. If there is a network model meeting the semantic analysis result, it indicates that the network model can meet the user demand, and at this time, there is no need to retrain a network model. Correspondingly, the network model meeting the semantic analysis result can be sent to the first terminal.

[0225] If there is no network model meeting the semantic analysis result, it indicates that the network model stored in the server currently does not meet the user demand and support deployment in the first terminal, and at this time, a new network model (i.e., a target network model) can be trained according to steps S203 to S204 described above, and the target network model is sent to the first terminal.

[0226] Based on the processing described above, in the case where there is a pre-trained network model meeting the user demand, the pre-trained network model can be directly sent to the first terminal, so that the efficiency of model deployment can be improved and the waste of computing resources of the server can be avoided.

[0227] In some embodiments, the electronic device can perform model optimization operations such as quantization and pruning on the target network model based on the terminal information of the first terminal and the structure information of the target network model. For example, the electronic device can call a quantization tool to quantize the target network model and call a pruning tool to prune the target network model.

[0228] Thus, the target network model obtained through training is optimized by the electronic device. For example, the size of the network model obtained through training can be reduced through model pruning without affecting the performance of the model, so that the pruned network model can be deployed in a terminal in which the network model before pruning cannot be deployed. Thus, the target network model can be deployed in the first terminal without the user manually optimizing the network model obtained through training, further simplifying the operation of the user, and enabling the user to more conveniently deploy the target network model.

[0229] In some embodiments, after obtaining the target network model through training, the target network model can also be verified.

[0230] In the embodiments of the present application, the electronic device can match the obtained hardware information of the first terminal with the performance requirement information of the target network model to determine whether the first terminal can run the target network model. If yes, the first terminal passes the verification, and if no, the first terminal fails the verification. The hardware information of the first terminal includes chip performance information of the first terminal, memory amount information of the first terminal, and the like.

[0231] Alternatively, the target network model can also be manually verified, that is, it is determined whether the target network model supports being deployed on the first terminal. If the target network model supports being deployed on the first terminal, the target network model is sent to the first terminal.

[0232] If the target network model fails the verification, the server can also send a response message prompting network model training exception to the first terminal.

[0233] Based on the above processing, the generated target network model can be reduced to be incompatible with the first terminal, and the user can be further helped to complete model deployment, further simplifying the operation of the user, and enabling the user to more conveniently deploy the target network model.

[0234] Referring to FIG. 7, FIG. 7 is a flowchart of model deployment provided by an embodiment of the present application. FIG. 7 includes the following steps:

[0235] S701: The user inputs generation requirement information through a smart device installed with a software application.

[0236] S702: The server receives and parses the generation requirement information input by the user.

[0237] That is, the electronic device (such as the server) can obtain the generation requirement information input by the user, and perform semantic analysis on the generation requirement information to obtain a semantic analysis result.

[0238] S703: The server determines whether the current network model meets the user's demand. If not, step S704 is executed; if yes, step S707 is executed.

[0239] S704: The server determines whether the training data determined by the terminal is sufficient. If yes, step S705 is executed; if not, step S706 is executed.

[0240] That is, the electronic device can determine the images in the first image library that match the first model information as first candidate images, and determine that the number of the first candidate images is greater than a preset threshold. The images stored in the first image library are historical images uploaded by the first terminal.

[0241] If yes, the training data corresponding to the terminal is sufficient; if not, the training data corresponding to the terminal is insufficient.

[0242] S705: Based on the training data corresponding to the terminal, a network model meeting the user's demand is trained.

[0243] That is, the first candidate image is determined as a training image, and the initial structure network model is trained based on the training image to obtain a target network model.

[0244] S706: Based on the images collected by other terminals in the same scene and third-party terminals in similar scenes, training data is determined, and a network model meeting the user's demand is trained.

[0245] That is, the electronic device can determine the images in the second image library that match the first model information as second candidate images. In addition, the images in the preset public image library that match the first model information are determined as third candidate images.

[0246] The images stored in the second image library are historical images uploaded by the second terminal. The second terminal is another terminal associated with the user to which the first terminal belongs, except for the first terminal. The images collected by the first terminal and the second terminal are consistent in scene. The images in the preset public image library (i.e., the images collected by third-party terminals in similar scenes) are consistent in scene with the images in the first image library.

[0247] Correspondingly, the electronic device can determine the first candidate image, the second candidate image, and the third candidate image as training images. The initial structure network model is trained based on the training images to obtain a target network model.

[0248] S707: The network model meeting the user's demand is issued to the first terminal.

[0249] That is, the electronic device can send the target network model trained to the first terminal, or send the network model currently possessed and meeting the user demand to the first terminal.

[0250] In the technical solution of the present application, the operations of image acquisition, storage, use, processing, transmission, provision and disclosure are all performed after obtaining the authorization of the user.

[0251] Based on the same inventive concept, the present application provides a model deployment system, which comprises a first terminal and a server.

[0252] The first terminal is configured to acquire generation requirement information of a network model to be deployed in the first terminal input by a user, and send the generation requirement information to the server.

[0253] The server is configured to execute the model deployment method described above.

[0254] The first terminal is further configured to receive a network model from the server.

[0255] Based on the same inventive concept, the present application provides a model deployment device, as shown in FIG. 8, which is a structural diagram of a model deployment device provided by an embodiment of the present application. The device comprises:

[0256] The generation requirement information acquisition module 801 is configured to acquire generation requirement information of a network model to be deployed in the first terminal input by a user.

[0257] The semantic analysis module 802 is configured to perform semantic analysis on the generation requirement information to obtain a semantic analysis result, wherein the semantic analysis result comprises first model information representing an identified subject of the network model to be deployed.

[0258] The first retrieval module 803 is configured to perform retrieval in a preset image library based on the first model information to obtain training images containing the identified subject, wherein the preset image library comprises a first image library for storing historical images uploaded by the first terminal, and / or a second image library for storing historical images uploaded by a second terminal associated with the user and other than the first terminal.

[0259] The training module 804 is configured to train an initial structure network model based on the training images to obtain a target network model, and send the target network model to the first terminal.

[0260] In an optional embodiment, the method is applied to a server, and the server is deployed with a large language model.

[0261] The semantic analysis module 802, the first retrieval module 803, and the training module 804 are specifically configured to:

[0262] The generated demand information is input into the large language model, semantic analysis of the generated demand information is performed by the large language model to obtain a semantic analysis result, a training image containing the identified subject is obtained by searching the first model information in a preset image library, and the target network model is obtained by training the initial structure network model based on the training image, and the first terminal is sent.

[0263] In an optional embodiment, the first retrieval module 803 comprises:

[0264] The image in the first image library that matches the first model information is determined as a first candidate image.

[0265] If the number of the first candidate images is greater than a preset threshold, the first candidate images are determined as training images.

[0266] In an optional embodiment, the apparatus further comprises:

[0267] The second candidate image determination module is configured to, if the number of the first candidate images is not greater than the preset threshold, determine the image in the second image library that matches the first model information as a second candidate image.

[0268] The first training image determination module is configured to, if the sum of the number of the first candidate images and the second candidate images is greater than the preset threshold, determine the first candidate images and the second candidate images as training images.

[0269] In an optional embodiment, the apparatus further comprises:

[0270] The third candidate image determination module is configured to, if the sum of the number of the first candidate images and the second candidate images is not greater than the preset threshold, determine the image in a preset public image library that matches the first model information as a third candidate image.

[0271] The second training image determination module is configured to determine the first candidate images, the second candidate images, and the third candidate images as training images.

[0272] In an optional embodiment, the images in the preset public image library are consistent with the scenes of the images in the first image library.

[0273] In an optional embodiment, for any image library, the image in the image library that matches the first model information is determined by the following steps:

[0274] determining, from the image library, an image with the subject label as a labeled image;

[0275] from the labeled image, determining an image in which the subject label contains the identified subject as a first matching image;

[0276] based on the text encoder in the pre-trained image-text retrieval model, performing feature extraction on the first model information to obtain text features of the first model information;

[0277] determining an unlabeled image in the image library that matches the text features as a second matching image; wherein the image features of any unlabeled image in the image library are obtained by performing feature extraction on the unlabeled image based on the image encoder in the image-text retrieval model;

[0278] combining the first matching image and the second matching image to obtain an image in the image library that matches the first model information.

[0279] In an optional embodiment, before searching in the preset image library based on the first model information to obtain a training image containing the identified subject, the method further comprises:

[0280] obtaining a reference image input by a user for the network model to be deployed;

[0281] The step of determining the image in the image library that matches the first model information further comprises:

[0282] performing feature extraction on the reference image to obtain image features of the reference image;

[0283] determining an image in the image library that matches the image features of the reference image as a third matching image; wherein the image in the image library that matches the first model information further includes the third matching image.

[0284] In an optional embodiment, the apparatus further comprises:

[0285] a first judgment module, configured to, before searching in the preset image library based on the first model information to obtain a training image containing the identified subject, judge whether there is a network model that meets the semantic analysis result;

[0286] a first delivery module, configured to, if there is a network model that meets the semantic analysis result, send the network model that meets the semantic analysis result to the first terminal;

[0287] The first retrieval module 803 is specifically configured to:

[0288] If there is no network model satisfying the semantic analysis result, a preset image library is searched based on the first model information to obtain a training image containing the recognition subject.

[0289] In an optional embodiment, the semantic analysis result further contains second model information representing a task type of the network model to be deployed.

[0290] The apparatus further includes:

[0291] The initial network model determination module is configured to, before training an initial structure network model based on the training image to obtain a target network model, determine a network model supporting deployment on the first terminal and having a task type consistent with the task type represented by the second model information as the initial structure network model according to terminal information of the first terminal.

[0292] In an optional embodiment, the generation requirement information contains terminal information of the first terminal; or the terminal information of the first terminal is queried from pre-recorded terminal information of each terminal.

[0293] In an optional embodiment, the first terminal and the second terminal collect images of consistent scenes.

[0294] Embodiments of the present application also provide an electronic device, as shown in FIG. 9, which includes:

[0295] The memory 901 is configured to store a computer program.

[0296] The processor 902 is configured to execute the program stored in the memory 901 to implement the steps of any of the above model deployment methods.

[0297] The above electronic device can further include a communication bus and / or a communication interface, and the processor 902, the communication interface and the memory 901 can communicate with each other through the communication bus.

[0298] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, only one thick line is shown in the figure, but it does not mean that there is only one bus or only one type of bus.

[0299] The communication interface is configured to communicate between the above electronic device and other devices.

[0300] The memory can include a random access memory (RAM) and can also include a non-volatile memory (NVM), such as at least one disk memory. In an optional embodiment, the memory can also be at least one storage device located remotely from the aforementioned processor.

[0301] The processor described above can be a general processor, including a central processing unit (CPU), a network processor (NP), etc.; can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic component, a discrete hardware component.

[0302] In yet another embodiment provided in the present application, a non-transitory computer readable storage medium is also provided, and the non-transitory computer readable storage medium stores a computer program. The computer program is executed by a processor to implement the steps of any of the model deployment methods described above.

[0303] In yet another embodiment provided in the present application, a computer program product containing instructions, which, when run on a computer, causes the computer to execute any of the model deployment methods in the above embodiments.

[0304] In the embodiments described above, all or part of the embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a non-transitory computer readable storage medium, or transferred from one non-transitory computer readable storage medium to another non-transitory computer readable storage medium, for example, the computer instructions can be transferred from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode. The non-transitory computer readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media. The available media can be magnetic media (for example, floppy disk, hard disk, magnetic tape), optical media (for example, DVD), or solid state disk (SSD) and the like.

[0305] It should be noted that, in this paper, the relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitation, the element defined by the statement "including a" does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0306] Each embodiment in the specification is described in a related manner, and the same and similar parts between each embodiment can be referred to each other, and each embodiment focuses on the difference from other embodiments. Especially, for system, device, electronic device, non-transitory computer readable storage medium embodiments, since they are basically similar to method embodiments, the description is relatively simple, and the related parts can refer to the part of the method embodiment.

[0307] The above description is only the preferred embodiment of the present application, and is not used to limit the present application, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of protection of the present application.

Claims

1. A model deployment method, the method comprising: obtaining user inputted generation requirement information of a network model to be deployed in a first terminal; performing semantic analysis on the generation requirement information to obtain a semantic analysis result; wherein the semantic analysis result contains first model information representing an identified subject of the network model to be deployed; based on the first model information, performing retrieval in a preset image library to obtain training images containing the identified subject; wherein the preset image library contains: a first image library for storing historical images uploaded by the first terminal, and / or a second image library for storing historical images uploaded by a second terminal associated with the user and other than the first terminal; training an initial structure network model based on the training images to obtain a target network model, and sending the target network model to the first terminal.

2. The method of claim 1, wherein, The method is applied to a server, and a large language model is deployed in the server; performing semantic analysis on the generation requirement information to obtain a semantic analysis result; based on the first model information, performing retrieval in a preset image library to obtain training images containing the identified subject; training an initial structure network model based on the training images to obtain a target network model, and sending the target network model to the first terminal, comprising: inputting the generation requirement information into the large language model to perform semantic analysis on the generation requirement information by the large language model to obtain a semantic analysis result; based on the first model information, performing retrieval in a preset image library to obtain training images containing the identified subject; training an initial structure network model based on the training images to obtain a target network model, and sending the target network model to the first terminal.

3. The method of claim 1 or 2, wherein, The based on the first model information, performing retrieval in a preset image library to obtain training images containing the identified subject, comprises: determining images in the first image library that match the first model information as first candidate images; if the number of first candidate images is greater than a preset threshold, the first candidate images are determined as training images.

4. The method of claim 3, wherein, The method further comprises: if the number of first candidate images is not greater than the preset threshold, determining images in the second image library that match the first model information as second candidate images; if the sum of the number of first candidate images and the number of second candidate images is greater than the preset threshold, the first candidate images and the second candidate images are determined as training images.

5. The method of claim 4, wherein, The method further comprises: if the sum of the number of first candidate images and the number of second candidate images is not greater than the preset threshold, determining images in a preset public image library that match the first model information as third candidate images; determining the first candidate images, the second candidate images, and the third candidate images as training images.

6. The method of claim 5, wherein, The images in the preset public image library are consistent with the scenes of the images in the first image library.

7. The method of claim 5, wherein, For any image library, the images in the image library that match the first model information are determined by the following steps: determine images with subject labels from the image library as labeled images; determining, from the tagged image, a subject label containing an image of the identified subject as a first matching image; performing feature extraction on the first model information based on a text encoder in a pre-trained image-text retrieval model to obtain text features of the first model information; determining an untagged image in the image library that has image features matching the text features as a second matching image; wherein the image features of any untagged image in the image library are obtained by performing feature extraction on the untagged image based on an image encoder in the image-text retrieval model; combining the first matching image and the second matching image to obtain an image in the image library that matches the first model information.

8. The method of claim 7, wherein, Before performing retrieval in the preset image library based on the first model information to obtain a training image containing the identified subject, the method further comprises: obtaining a reference image input by a user for the network model to be deployed; the step of determining the image in the image library that matches the first model information further comprises: performing feature extraction on the reference image to obtain image features of the reference image; determining an image in the image library that has image features matching the image features of the reference image as a third matching image; wherein the image in the image library that matches the first model information further includes the third matching image.

9. The method of claim 1, wherein, Before performing retrieval in the preset image library based on the first model information to obtain a training image containing the identified subject, the method further comprises: determining whether there is a network model that meets the semantic analysis result at present; if there is a network model that meets the semantic analysis result, sending the network model that meets the semantic analysis result to the first terminal; the step of performing retrieval in the preset image library based on the first model information to obtain a training image containing the identified subject comprises: if there is no network model that meets the semantic analysis result, performing retrieval in the preset image library based on the first model information to obtain a training image containing the identified subject.

10. The method of claim 1, wherein, the semantic analysis result further contains second model information representing a task type of the network model to be deployed; before training an initial structure network model based on the training image to obtain a target network model, the method further comprises: determining, according to terminal information of the first terminal, a network model that supports deployment in the first terminal and has a task type consistent with the task type represented by the second model information as an initial structure network model.

11. The method of claim 10, wherein, the generation requirement information contains terminal information of the first terminal; or, the terminal information of the first terminal is queried from pre-recorded terminal information of each terminal.

12. The method of claim 1, wherein, the images collected by the first terminal and the second terminal are consistent in scene.

13. A model deployment system, the system comprising: a first terminal and a server; the first terminal is configured to obtain generation requirement information of a network model to be deployed in the first terminal input by a user, and send the generation requirement information to the server; the server is configured to execute the model deployment method in any one of claims 1-12; The first terminal is further configured to receive the network model from the server. 14.A model deployment apparatus, comprising: a generation requirement information obtaining module configured to obtain generation requirement information of a network model to be deployed in a first terminal input by a user; a semantic analysis module configured to perform semantic analysis on the generation requirement information to obtain a semantic analysis result, wherein the semantic analysis result comprises first model information representing an identified subject of the network model to be deployed; a first searching module configured to search in a preset image library based on the first model information to obtain training images containing the identified subject, wherein the preset image library comprises a first image library configured to store historical images uploaded by the first terminal and / or a second image library configured to store historical images uploaded by a second terminal associated with the user and different from the first terminal; a training module configured to train an initial structure network model based on the training images to obtain a target network model and send the target network model to the first terminal. 15.An electronic device, comprising: a memory configured to store a computer program; a processor configured to execute the program stored in the memory to implement the method of any one of claims 1-12. 16.A non-transitory computer readable storage medium, wherein the non-transitory computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method of any one of claims 1-12. 17.A computer program product comprising instructions which, when executed on a computer, cause the computer to carry out the steps of the method of any one of the preceding claims 1-12.

Images