Visual control method and device based on multiple models and electronic equipment

Abstract

The invention provides a multi-model-based visual control method and device, electronic equipment and a readable storage medium, and relates to the technical field of artificial intelligence such as computer vision, large models, deep learning and cloud services. The multi-model-based visual control method comprises the steps of obtaining a to-be-processed image and target task information corresponding to the to-be-processed image; obtaining a first target model according to the target task information, and obtaining a task perception result according to the to-be-processed image by using the first target model; obtaining a second target model according to the target task information, and obtaining a task reasoning result according to the to-be-processed image and the task perception result by using the second target model; and in response to determining that the task reasoning result meets a preset requirement, controlling a target device to execute a target operation. The flexibility and accuracy of visual control can be improved, and the cost of visual control is reduced.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer, in particular to the technical field of computer vision, large model, deep learning, cloud service and artificial intelligence. A multi-model-based visual control method and device, electronic equipment and readable storage medium are provided. BACKGROUND

[0002] The prior art usually adopts two ways when performing visual control on a device: 1) using a special small model corresponding to different tasks to perform task reasoning, so as to control the device to perform corresponding operations according to the task reasoning result; 2) using a multi-model large model to perform task reasoning, so as to control the device to perform corresponding operations according to the task reasoning result.

[0003] When using a special small model to perform task reasoning, only one specific task reasoning can be completed, the generalization ability is poor, and a large amount of high-quality data needs to be obtained to train the special small model; while using a multi-modal large model to perform task reasoning, the cost is high, and the accuracy of the task reasoning result obtained by the multi-modal large model is not as good as that of the special small model. SUMMARY

[0004] According to a first aspect of the present disclosure, a multi-model-based visual control method is provided, comprising: obtaining a to-be-processed image and target task information corresponding to the to-be-processed image; obtaining a first target model according to the target task information, using the first target model to obtain a task perception result according to the to-be-processed image; obtaining a second target model according to the target task information, using the second target model to obtain a task reasoning result according to the to-be-processed image and the task perception result; and in response to determining that the task reasoning result meets a preset requirement, controlling a target device to perform a target operation.

[0005] According to a second aspect of the present disclosure, a multi-model-based visual control device is provided, comprising: an obtaining unit configured to obtain a to-be-processed image and target task information corresponding to the to-be-processed image; a perception unit configured to obtain a first target model according to the target task information, and use the first target model to obtain a task perception result according to the to-be-processed image; a reasoning unit configured to obtain a second target model according to the target task information, and use the second target model to obtain a task reasoning result according to the to-be-processed image and the task perception result; and a control unit configured to control a target device to perform a target operation in response to determining that the task reasoning result meets a preset requirement.

[0006] According to a third aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method as described above.

[0007] According to a fourth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform the method described above.

[0008] According to a fifth aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the method described above.

[0009] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0010] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0011] Figure 1 This is a schematic diagram based on the first embodiment of the present disclosure;

[0012] Figure 2 This is a schematic diagram according to the second embodiment of the present disclosure;

[0013] Figure 3 This is a schematic diagram according to the third embodiment of the present disclosure;

[0014] Figure 4 This is a schematic diagram according to the fourth embodiment of the present disclosure;

[0015] Figure 5 This is a schematic diagram according to the fifth embodiment of the present disclosure;

[0016] Figure 6 This is a schematic diagram according to the sixth embodiment of the present disclosure;

[0017] Figure 7 This is a block diagram of an electronic device used to implement the multi-model-based vision control method of the embodiments of this disclosure. Detailed Implementation

[0018] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and mechanisms are omitted in the following description.

[0019] Figure 1 This is a schematic diagram based on the first embodiment of this disclosure. (See diagram below.) Figure 1 As shown, the multi-model-based visual control method in this embodiment specifically includes the following steps:

[0020] S101. Obtain the image to be processed and the target task information corresponding to the image to be processed;

[0021] S102. Obtain a first target model based on the target task information, and use the first target model to obtain a task perception result based on the image to be processed.

[0022] S103. Obtain a second target model based on the target task information, and use the second target model to obtain a task reasoning result based on the image to be processed and the task perception result.

[0023] S104. In response to determining that the task reasoning result meets the preset requirements, control the target device to perform the target operation.

[0024] The multi-model-based vision control method in this embodiment decouples the vision control process into a task perception process and a task reasoning process. The first target model, obtained based on the target task information, performs task perception, and the second target model, also obtained based on the target task information, performs task reasoning. Based on the obtained task reasoning results, the target device is controlled to perform the corresponding target operation. This embodiment can improve the flexibility and accuracy of vision control and reduce the cost of vision control.

[0025] The execution subject of the multi-model-based vision control method in this embodiment is the target device. The target device can be an image acquisition device or a device that includes an image acquisition device, such as a vehicle that includes an image acquisition device or an alarm device that includes an image acquisition device.

[0026] In this embodiment, when executing S101, the image acquired in real time by the image acquisition device can be used as the image to be processed.

[0027] In this embodiment, after executing S101 to acquire the image to be processed, preset task information corresponding to the target device can be acquired as the target task information for the image to be processed; wherein, the preset task information corresponding to the target device can be one or multiple.

[0028] For example, the target task information obtained by executing S101 in this embodiment can be "whether there is anyone in the site", "whether the refrigerator door in the site is open", "whether the worker is sitting inside the coal mining machine", "whether the worker is wearing a safety helmet", etc.

[0029] In other words, this embodiment will pre-set the task information of the corresponding target device, so that after the target device is deployed, it can automatically execute the task corresponding to the preset task information, such as personnel detection task, refrigerator door opening detection task, etc.

[0030] In this embodiment, after executing S101 to obtain the image to be processed, real-time task information (such as task information input to the target device in real time or task information generated by the target device in real time) can also be obtained as the target task information for the corresponding image to be processed.

[0031] For example, the target task information obtained by executing S101 in this embodiment can be "whether there are other vehicles in front of the vehicle" or "whether there are pedestrians in front of the vehicle" generated in real time during the autonomous driving process.

[0032] In other words, this embodiment can also acquire task information generated in real time by the target device during operation (e.g., a vehicle with autonomous driving capabilities during driving) or task information input to the target device in real time, as target task information, thereby expanding the application scenarios of vision control.

[0033] In this embodiment, after executing S101 to obtain the image to be processed, preset task information and real-time task information can also be obtained simultaneously as target task information for the corresponding image to be processed.

[0034] For example, if the preset task information for the target device is only "whether there is anyone in the venue", in this embodiment, when executing S101, in addition to obtaining "whether there is anyone in the venue" as the target task information, it can also obtain "whether the refrigerator door in the venue is open" which is input to the target device in real time as the target task information.

[0035] In other words, in addition to having the ability to "automatically execute tasks corresponding to preset task information", the target device in this embodiment also has the ability to "execute tasks corresponding to real-time task information", thereby greatly enhancing the task execution capability of the target device and expanding the application scenarios of vision control.

[0036] In this embodiment, after executing S101 to obtain the image to be processed and its corresponding target task information, S102 is executed to obtain the first target model based on the target task information, and the first target model is used to obtain the task perception result based on the image to be processed.

[0037] In this embodiment, the first target model obtained in step S102 is a visual expert model. Different visual expert models are used to perceive different types of objects in an image, such as a visual expert model for perceiving "people" in an image, a visual expert model for perceiving "vehicles" in an image, and a visual expert model for perceiving "refrigerators" in an image.

[0038] Therefore, when executing S102 to obtain the first target model based on the target task information, this embodiment can adopt the following implementation method: determine the target perception object based on the target task information. In this embodiment, the target perception object corresponding to the target task information can be determined based on the preset correspondence between the task information and the perception object; obtain the visual expert model corresponding to the determined target perception object as the first target model.

[0039] In other words, this embodiment obtains a first target model based on the target perception object determined by the target task information, so that the obtained first target model can perceive the target perception object in the image to be processed, thereby improving the accuracy of obtaining the first target model and thus improving the accuracy of the task perception result obtained by using the first target model.

[0040] In this embodiment, the first target model corresponding to the target perception object is located locally on the target device. That is, in this embodiment, a visual expert model for perceiving a specific object is pre-deployed on the target device, which can improve the acquisition speed of the first target model and thus improve the acquisition speed of the task perception result.

[0041] In this embodiment, after executing S102 to obtain the first target model corresponding to the target task information, the image to be processed is input into the first target model, and then the task perception result corresponding to the target task information is obtained based on the output result of the first target model.

[0042] The task perception result obtained by executing S102 in this embodiment can be standardized structural information such as whether the image to be processed includes the target perception object and the location information of the target perception object in the image to be processed.

[0043] In addition, when executing S102, this embodiment may also include the following: in response to determining that the first target model could not be obtained based on the target task information, sending the image to be processed and the target task information to the server, so as to use the multimodal large model located on the server to obtain the task inference result based on the image to be processed and the target task information; receiving the task inference result returned by the server.

[0044] In other words, in this embodiment, if the first target model cannot be obtained based on the target task information, it indicates that the obtained target task information is new task information. The visual expert model located on the target device cannot perceive the perception object corresponding to the target task information. Therefore, through the interaction between the device (i.e., the target device) and the cloud (i.e., the server), the multimodal large model located in the cloud is used to obtain the task perception result, ensuring that the purpose of controlling the target device can be achieved even when the received target task information is new task information.

[0045] In this embodiment, a multimodal large model is an artificial intelligence model that can simultaneously process and understand multiple types of data (such as text, images, audio, video, etc.). In this embodiment, the multimodal large model can be a multimodal large language model (MLLM). Due to the large number of parameters, multimodal large models are usually deployed in the cloud.

[0046] In this embodiment, after obtaining the task perception result by executing S102, S103 is executed to obtain the second target model based on the target task information. Using the second target model, the task inference result is obtained based on the image to be processed and the task perception result.

[0047] In this embodiment, the second target model obtained in S103 is a scenario-based visual large model. Different scenario-based visual large models are used to perform different scenario-based task reasoning on images. For example, scenario-based visual large models are used for task reasoning in security scenarios (e.g., reasoning whether an outsider has entered the site, reasoning whether the refrigerator door in the site is open, etc.), scenario-based visual large models are used for task reasoning in autonomous driving scenarios (e.g., reasoning whether the vehicle can accelerate, reasoning whether the vehicle can change lanes, etc.), and scenario-based visual large models are used for task reasoning in industrial and mining scenarios (e.g., reasoning whether the worker is wearing a safety helmet, reasoning whether the worker is sitting inside the coal mining machine, reasoning whether the worker has entered a dangerous area in the construction site, etc.).

[0048] Therefore, when executing S103 to obtain the second target model based on the target task information, this embodiment can adopt the following implementation method: determine the target reasoning scenario based on the target task information. In this embodiment, the target reasoning scenario corresponding to the target task information can be determined based on the preset correspondence between the task information and the reasoning scenario; obtain the scene-based visual large model corresponding to the determined target reasoning scenario as the second target model.

[0049] In other words, this embodiment obtains a second target model based on the target reasoning scenario determined by the target task information, so that the obtained second target model can perform task reasoning in the corresponding target reasoning scenario, thereby improving the accuracy of obtaining the second target model and thus improving the accuracy of the task reasoning results obtained by using the second target model.

[0050] In this embodiment, the second target model corresponding to the target reasoning scenario can be located locally on the target device. That is, in this embodiment, a scenario-based visual large model for task reasoning in a specific scenario will be pre-deployed on the target device, which can improve the acquisition speed of the second target model and thus improve the acquisition speed of the task reasoning result.

[0051] In addition, since the scene-based visual large model has more parameters than the visual expert model, in order to make more effective use of the target device's storage resources, the second target model corresponding to the target reasoning scene in this embodiment can also be located on the server (i.e., the cloud). That is, when executing S103 in this embodiment, the image to be processed and the task perception result can be sent to the server so as to use the second target model in the server to obtain the task reasoning result based on the received information.

[0052] In addition, when executing S103, this embodiment may also include the following: in response to determining that the task inference result cannot be obtained according to the second target model, the image to be processed and the task perception result are sent to the server so as to use the multimodal large model located on the server to obtain the task inference result according to the image to be processed and the task perception result; and the task inference result returned by the server is received.

[0053] In other words, if the task reasoning result cannot be obtained from the second target model in this embodiment, it indicates that the second target model cannot reason based on the currently obtained task perception result. The target task information is new task information belonging to the target reasoning scenario. Thus, through the interaction between the device (i.e., the target device) and the cloud (i.e., the server), the multimodal large model located in the cloud is used to obtain the task perception result, ensuring that the purpose of controlling the target device can still be achieved even if the second target model cannot obtain the task reasoning result.

[0054] Since this embodiment obtains the second target model by reasoning the scenario, and the same reasoning scenario may include multiple tasks, when different target task information corresponds to the same task reasoning scenario, this embodiment only needs to obtain one second target model when executing S103, and can obtain the task reasoning results corresponding to different task information respectively. This can greatly reduce the number of scenario-based visual large models required, thereby reducing the cost of visual control.

[0055] For example, if the target task information includes task information 1 (whether there is anyone in the venue) and task information 2 (whether the refrigerator door in the venue is open), and if the reasoning scenario corresponding to the above two target task information is a "security scenario", then after obtaining the task perception result of the corresponding task information 1 and the task perception result of the corresponding task information 2, this embodiment can use the second target model corresponding to the "security scenario" to obtain the task reasoning result of the corresponding task information 1 (whether there is someone in the venue or no one in the venue) and the task reasoning result of the corresponding task information 2 (whether the refrigerator door in the venue is open or the refrigerator door in the venue is closed).

[0056] In this embodiment, after obtaining the task reasoning result by executing S103, S104 is executed in response to determining that the task reasoning result meets the preset requirements, and the target device is controlled to perform the target operation.

[0057] In this embodiment, when executing S104, the operation requirements and their corresponding target operations can be obtained first based on the target task information. Then, in response to determining that the obtained task reasoning result meets the operation requirements, the target device is controlled to execute the target operation.

[0058] For example, if the target task information is "whether there is anyone in the venue", and the operation requirement corresponding to the target task information is "if there is someone in the venue, trigger an alarm", then the target operation corresponding to the operation requirement is "alarm". If the obtained task reasoning result is "there is someone in the venue", then in this embodiment, when executing S104, the target device can be controlled to trigger an alarm.

[0059] In other words, this embodiment obtains the task reasoning results of the corresponding target task information by using the first target model and the second target model in sequence, and then controls the target device based on the obtained task reasoning results. Since this embodiment decouples the visual control process into the task perception process and the task reasoning process, and the same scene-based visual large model can complete the reasoning of different tasks in its corresponding reasoning scenario, this embodiment does not need to set different visual large models for different tasks, which reduces the model acquisition cost and improves the flexibility when performing visual control on the target device.

[0060] Figure 2 This is a schematic diagram according to the second embodiment of this disclosure. (See diagram below.) Figure 2 As shown in the figure, the first target model corresponding to the target perception object in this embodiment can be trained in the following way:

[0061] S201. Obtain a first training set, which includes multiple first sample images and the perception annotation results of the corresponding target perception objects in the first sample images;

[0062] S202. Input the first sample image into the first neural network model to obtain the perception prediction result output by the first neural network model;

[0063] S203. Calculate a first loss function value based on the perception prediction results and perception annotation results of the plurality of first sample images, and use the first loss function value to adjust the parameters of the first neural network model to obtain the first target model.

[0064] In other words, this embodiment uses the first sample image and the perception annotation results of the corresponding target perception object in the first sample image to complete the training of the first neural network model to obtain the first target model, so that the trained first target model can obtain the task perception result of the corresponding target perception object in the image based on the input image.

[0065] The first neural network model in this embodiment can be a convolutional neural network model, a recurrent neural network model, or a multimodal large model.

[0066] In this embodiment, after obtaining the first target model by executing S203, the obtained first target model can be deployed to the target device locally, so that the target device can use the first target model located locally to quickly obtain task perception results.

[0067] Figure 3 This is a schematic diagram according to the third embodiment of this disclosure. (See diagram below.) Figure 3 As shown in the figure, the second target model corresponding to the target reasoning scenario in this embodiment can be trained in the following way:

[0068] S301. Obtain a second training set, which includes multiple second sample images and perception annotation results and inference annotation results of the target inference scene corresponding to the second sample images;

[0069] S302. Input the second sample image and its corresponding perceptual annotation results into the second neural network model to obtain the inference prediction results output by the second neural network model;

[0070] S303. Calculate the second loss function value based on the inference prediction results and inference annotation results of the plurality of second sample images, and use the second loss function value to adjust the parameters of the second neural network model to obtain the second target model.

[0071] In other words, this embodiment uses the second sample image and the perception annotation results and reasoning annotation results of the corresponding target reasoning scene in the second sample image to complete the training of the second neural network model to obtain the second target model, so that the trained second target model can obtain the task reasoning result of the corresponding target reasoning scene in the image based on the input image and the perception annotation results in the image.

[0072] The second neural network model in this embodiment can be a convolutional neural network model, a recurrent neural network model, or a multimodal large model.

[0073] In this embodiment, after obtaining the second target model by executing S303, the obtained second target model can be deployed to the target device locally, so that the target device can use the second target model located locally to quickly obtain the task inference results.

[0074] It is understood that when executing S301 in this embodiment, at least one target task information can be determined according to the target reasoning scenario, and then the perception annotation results and reasoning annotation results corresponding to different target task information in the second sample image can be obtained according to the determined at least one target task information, so as to use them as training samples in the second training set to train the second neural network model.

[0075] Figure 4 This is a schematic diagram according to the fourth embodiment of this disclosure. (See diagram below.) Figure 4 As shown, this embodiment may also include the following:

[0076] S401. After sending the image to be processed and the target task information to the server, receive the first target model corresponding to the target task information returned by the server. The first target model is obtained by training the multimodal large model in the server using the first labeled data corresponding to the target task information.

[0077] S402. Deploy the first target model locally on the target device.

[0078] In other words, in this embodiment, when the image to be processed and the target task information are sent to the server, it indicates that the target task information is new task information. The existing visual expert model in the target device cannot complete the object perception corresponding to the target task information. Therefore, after the image to be processed and the target task information are sent to the server, the multimodal large model in the server can obtain the first annotation data based on the received image and task information. Then, the obtained first annotation data is used to train the first target model corresponding to the target task information. Finally, the obtained first target model is sent to the target device so that the target device can deploy the received first target model locally. Thus, when the same task information as the target task information is obtained again, the first target model deployed locally can be used directly to perceive the target object in the image.

[0079] In this embodiment, the first labeled data corresponding to the target task information used by the multi-model large model can be the image to be processed and the task perception result corresponding to the target task information in the image to be processed (the task perception result is obtained by the multi-modal large model); in this embodiment, the multi-modal large model can train the first target model (i.e., the visual expert model) corresponding to the target task information after accumulating sufficient first labeled data.

[0080] It is understood that when executing S401 in this embodiment, a second target model corresponding to the target task information returned by the server can also be received, namely, a scene-based visual large model corresponding to the target task information, and then the received second target model can be deployed locally on the target device.

[0081] In other words, in this embodiment, when the target task information is new task information and there is no scene-based visual large model corresponding to the new task information, the multimodal large model can use the second labeled data of the corresponding target task information to train a second target model corresponding to the target task information.

[0082] The second labeled data corresponding to the target task information used in this embodiment can be the image to be processed, the task perception result and task reasoning result of the target task information in the image to be processed (the task reasoning result is obtained by the multimodal large model); the multimodal large model in this embodiment can train the second target model (i.e., the scene-based visual large model) corresponding to the target task information after accumulating sufficient second labeled data.

[0083] Figure 5 This is a schematic diagram according to the fifth embodiment of the present disclosure. Figure 5The diagram shows the structure of this embodiment when performing visual control based on multiple models: In this embodiment, the visual expert model (i.e., the first target model) corresponding to the target task information and the scene-based visual large model (i.e., the second target model) are deployed locally on the target device (i.e., the end side), and the multimodal large model is deployed on the server (i.e., the cloud side).

[0084] In this embodiment, after acquiring the image to be processed and the target task information, a first target model is first determined based on the target task information. If it exists, the image to be processed and the target task information are sent to the first target model to obtain the task perception result output by the first target model. Then, the image to be processed and the task perception result are sent to the second target model. If the task reasoning result output by the second target model is obtained, the target device is controlled to perform corresponding operations based on the obtained task reasoning result.

[0085] If there is no first target model with corresponding target task information, the image to be processed and the target task information are sent to the multimodal large model, and then the target device is controlled to perform corresponding operations based on the task inference results output by the multimodal large model.

[0086] If the task reasoning result cannot be obtained from the second target model, the image to be processed and the task perception result are sent to the multimodal large model, and then the target device is controlled to perform corresponding operations based on the task reasoning result output by the multimodal large model.

[0087] The multimodal large model located in the cloud can continuously collect labeled data corresponding to the received target task information during subsequent processing. After accumulating sufficient labeled data, it can train the first target model and the second target model corresponding to the target task information. Then, the trained first target model and the second target model are sent to the target device so that the target device can obtain the task inference result by using the local target model after receiving the target task information.

[0088] Figure 6 This is a schematic diagram according to the sixth embodiment of this disclosure. (See diagram below.) Figure 6 As shown, the multi-model-based vision control device 600 of this embodiment includes:

[0089] Acquisition unit 601 is used to acquire the image to be processed and the target task information corresponding to the image to be processed;

[0090] The perception unit 602 is used to obtain a first target model based on the target task information, and use the first target model to obtain a task perception result based on the image to be processed;

[0091] Inference unit 603 is used to obtain a second target model based on the target task information, and use the second target model to obtain a task inference result based on the image to be processed and the task perception result;

[0092] Control unit 604 is used to control the target device to perform the target operation in response to determining that the task reasoning result meets the preset requirements.

[0093] The multi-model-based vision control device in this embodiment is located on the target device, which can be an image acquisition device or a device that includes an image acquisition device, such as a vehicle that includes an image acquisition device or an alarm device that includes an image acquisition device.

[0094] The acquisition unit 601 can acquire images acquired in real time by the image acquisition device as images to be processed.

[0095] After acquiring the image to be processed, the acquisition unit 601 can acquire the preset task information corresponding to the target device as the target task information for the image to be processed; wherein, the preset task information corresponding to the target device can be one or multiple.

[0096] In other words, this embodiment will pre-set the task information of the corresponding target device, so that after the target device is deployed, it can automatically execute the task corresponding to the preset task information, such as personnel detection task, refrigerator door opening detection task, etc.

[0097] After acquiring the image to be processed, the acquisition unit 601 can also acquire real-time task information (such as task information input to the target device in real time or task information generated by the target device in real time) as the target task information for the corresponding image to be processed.

[0098] In other words, this embodiment can also acquire task information generated in real time by the target device during operation (e.g., a vehicle with autonomous driving capabilities during driving) or task information input to the target device in real time, as target task information, thereby expanding the application scenarios of vision control.

[0099] After acquiring the image to be processed, the acquisition unit 601 can also acquire preset task information and real-time task information at the same time, as the target task information for the corresponding image to be processed.

[0100] In other words, in addition to having the ability to "automatically execute tasks corresponding to preset task information", the target device in this embodiment also has the ability to "execute tasks corresponding to real-time task information", thereby greatly enhancing the task execution capability of the target device and expanding the application scenarios of vision control.

[0101] In this embodiment, after the acquisition unit 601 acquires the image to be processed and its corresponding target task information, the perception unit 602 acquires the first target model based on the target task information, and uses the first target model to obtain the task perception result based on the image to be processed.

[0102] The first target model acquired by the perception unit 602 is a visual expert model; different visual expert models are used to perceive different types of objects in the image.

[0103] Therefore, when the perception unit 602 obtains the first target model based on the target task information, the following implementation method can be adopted: determine the target perception object based on the target task information. In this embodiment, the target perception object corresponding to the target task information can be determined based on the preset correspondence between the task information and the perception object; obtain the visual expert model corresponding to the determined target perception object as the first target model.

[0104] In other words, the perception unit 602 obtains the first target model based on the target perception object determined by the target task information, so that the obtained first target model can perceive the target perception object in the image to be processed, thereby improving the accuracy of obtaining the first target model and thus improving the accuracy of the task perception result obtained by using the first target model.

[0105] In this embodiment, the first target model corresponding to the target perception object is located locally on the target device. That is, in this embodiment, a visual expert model for task perception is pre-deployed on the target device, which can improve the acquisition speed of the first target model and thus improve the acquisition speed of task perception results.

[0106] After acquiring the first target model corresponding to the target task information, the perception unit 602 inputs the image to be processed into the first target model, and then obtains the task perception result corresponding to the target task information based on the output result of the first target model.

[0107] The task perception result obtained by the perception unit 602 can be standardized structural information such as whether the image to be processed includes the target perception object and the location information of the target perception object in the image to be processed.

[0108] In addition, the perception unit 602 may also perform the following: in response to determining that the first target model could not be obtained based on the target task information, sending the image to be processed and the target task information to the server, so as to use the multimodal large model located on the server to obtain the task inference result based on the image to be processed and the target task information; and receiving the task inference result returned by the server.

[0109] In other words, if the perception unit 602 is unable to obtain the first target model based on the target task information, it indicates that the obtained target task information is new task information. The visual expert model located on the target device cannot perceive the perception object corresponding to the target task information. Therefore, through the interaction between the device end (i.e., the target device) and the cloud (i.e., the server), the multimodal large model located in the cloud is used to obtain the task perception result, ensuring that even if the received target task information is new task information, the purpose of controlling the target device can still be achieved.

[0110] In this embodiment, a multimodal large model is an artificial intelligence model that can simultaneously process and understand multiple types of data (such as text, images, audio, etc.). In this embodiment, the multimodal large model can be a multimodal large language model (MLLM). Due to the large number of parameters, multimodal large models are usually deployed in the cloud.

[0111] The multi-model-based vision control device 600 of this embodiment may further include a first training unit 605, which is used to pre-train a first target model in the following manner: acquiring a first training set, the first training set including multiple first sample images and perception annotation results of corresponding target perception objects in the first sample images; inputting the first sample images into a first neural network model to obtain the perception prediction results output by the first neural network model; calculating a first loss function value based on the perception prediction results and perception annotation results of the multiple first sample images; and using the first loss function value to adjust the parameters of the first neural network model to obtain the first target model.

[0112] In other words, the first training unit 605 uses the first sample image and the perception annotation results of the corresponding target perception object in the first sample image to complete the training of the first neural network model to obtain the first target model, so that the trained first target model can obtain the task perception result of the corresponding target perception object in the image based on the input image.

[0113] After obtaining the first target model, the first training unit 605 can deploy the obtained first target model to the target device locally, so that the target device can use the first target model located locally to quickly obtain task perception results.

[0114] In this embodiment, after the perception unit 602 obtains the task perception result, the reasoning unit 603 obtains the second target model based on the target task information, and uses the second target model to obtain the task reasoning result based on the image to be processed and the task perception result.

[0115] The second target model acquired by the reasoning unit 603 is a scene-based visual large model; different scene-based visual large models are used to perform task reasoning on images in different scenes.

[0116] Therefore, when the reasoning unit 603 obtains the second target model based on the target task information, the following implementation method can be adopted: Based on the target task information, determine the target reasoning scenario. In this embodiment, the target reasoning scenario corresponding to the target task information can be determined based on the correspondence between the preset task information and the reasoning scenario; obtain the scene-based visual large model corresponding to the determined target reasoning scenario as the second target model.

[0117] In other words, the reasoning unit 603 obtains the second target model based on the target reasoning scenario determined by the target task information, so that the obtained second target model can perform task reasoning in the corresponding target reasoning scenario, thereby improving the accuracy of obtaining the second target model and thus improving the accuracy of the task reasoning results obtained by using the second target model.

[0118] In this embodiment, the second target model corresponding to the target reasoning scenario can be located locally on the target device. That is, in this embodiment, a scenario-based visual large model for task reasoning in a specific scenario will be pre-deployed on the target device, which can improve the acquisition speed of the second target model and thus improve the acquisition speed of the task reasoning result.

[0119] In addition, since the scene-based visual large model has more parameters than the visual expert model, in order to make more effective use of the target device's storage resources, the second target model corresponding to the target inference scene in this embodiment can also be located on the server (i.e., the cloud). That is, the inference unit 603 can send the image to be processed and the task perception result to the server so as to use the second target model in the server to obtain the task inference result based on the received information.

[0120] In addition, the inference unit 603 may also perform the following: in response to determining that the task inference result cannot be obtained based on the second target model, send the image to be processed and the task perception result to the server, so as to use the multimodal large model located on the server to obtain the task inference result based on the image to be processed and the task perception result; and receive the task inference result returned by the server.

[0121] In other words, if the reasoning unit 603 is unable to obtain the task reasoning result based on the second target model, it indicates that the second target model cannot reason based on the currently obtained task perception result. Therefore, through the interaction between the device (i.e., the target device) and the cloud (i.e., the server), the multimodal large model located in the cloud is used to obtain the task perception result, ensuring that the purpose of controlling the target device can still be achieved even if the second target model cannot obtain the task reasoning result.

[0122] Since this embodiment obtains the second target model through reasoning scenarios, and the same reasoning scenario may include multiple tasks, when different target task information corresponds to the same task reasoning scenario, the reasoning unit 603 only needs to obtain one second target model to obtain the task reasoning results corresponding to different task information. This can greatly reduce the number of scenario-based visual large models required, thereby reducing the cost of visual control.

[0123] The multi-model-based vision control device 600 of this embodiment may further include a second training unit 606, used to pre-train a second target model in the following manner: acquiring a second training set, the second training set including multiple second sample images and perceptual annotation results and inference annotation results of the target inference scene corresponding to the second sample images; inputting the second sample images and their corresponding perceptual annotation results into a second neural network model to obtain the inference prediction results output by the second neural network model; calculating a second loss function value based on the inference prediction results and inference annotation results of the multiple second sample images; and using the second loss function value to adjust the parameters of the second neural network model to obtain the second target model.

[0124] In other words, the second training unit 606 uses the second sample image and the perception annotation results and reasoning annotation results of the corresponding target reasoning scene in the second sample image to complete the training of the second neural network model to obtain the second target model, so that the trained second target model can obtain the task reasoning result of the corresponding target reasoning scene in the image based on the input image and the perception annotation results in the image.

[0125] After obtaining the second target model, the second training unit 606 can deploy the obtained second target model to the target device locally, so that the target device can use the second target model located locally to quickly obtain the task inference results.

[0126] In this embodiment, after the reasoning unit 603 obtains the task reasoning result, the control unit 604 responds by determining that the task reasoning result meets the preset requirements and controls the target device to perform the target operation.

[0127] The control unit 604 can first obtain the operation requirements and their corresponding target operations based on the target task information, and then, in response to determining that the obtained task reasoning result meets the operation requirements, control the target device to execute the target operation.

[0128] In other words, this embodiment obtains the task reasoning results of the corresponding target task information by using the first target model and the second target model in sequence, and then controls the target device based on the obtained task reasoning results. Since this embodiment decouples the visual control process into the task perception process and the task reasoning process, and the same scene-based visual large model can complete the reasoning of different tasks in its corresponding reasoning scenario, this embodiment does not need to set different visual large models for different tasks, which reduces the cost of obtaining the reasoning model and also improves the flexibility when visually controlling the target device.

[0129] The multi-model-based vision control device 600 of this embodiment may further include a deployment unit 607, which is used to perform the following: after sending the image to be processed and the target task information to the server, receiving the first target model corresponding to the target task information returned by the server, wherein the first target model is obtained by training the multimodal large model in the server using the first labeled data of the corresponding target task information; and deploying the first target model locally on the target device.

[0130] In other words, when the image to be processed and the target task information are sent to the server in this embodiment, it indicates that the target task information is new task information. The existing visual expert model in the target device cannot complete the object perception corresponding to the target task information. Therefore, after the image to be processed and the target task information are sent to the server, the multimodal large model in the server can obtain labeled data based on the received image and task information, and then use the obtained labeled data to train the first target model corresponding to the target task information. Finally, the obtained first target model is sent to the target device so that the target device can deploy the received first target model locally. Thus, when the same task information as the target task information is obtained again, the first target model deployed locally can be used directly to complete the perception of the target object.

[0131] In this embodiment, the first labeled data corresponding to the target task information used by the multi-model large model can be the image to be processed and the task perception result corresponding to the target task information in the image to be processed (the task perception result is obtained by the multi-modal large model); in this embodiment, the multi-modal large model can train the first target model (i.e., the visual expert model) corresponding to the target task information after accumulating sufficient first labeled data.

[0132] It is understandable that the deployment unit 607 can also receive the second target model corresponding to the target task information returned by the server, that is, the scene-based visual large model corresponding to the target task information, and then deploy the received second target model locally on the target device.

[0133] In other words, in this embodiment, when the target task information is new task information and there is no scene-based visual large model corresponding to the new task information, the multimodal large model can use the second labeled data of the corresponding target task information to train a second target model corresponding to the target task information.

[0134] The second labeled data corresponding to the target task information used in this embodiment can be the image to be processed, the task perception result and task reasoning result of the target task information in the image to be processed (the task reasoning result is obtained by the multimodal large model); the multimodal large model in this embodiment can train the second target model (i.e., the scene-based visual large model) corresponding to the target task information after accumulating sufficient second labeled data.

[0135] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0136] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0137] like Figure 7 The diagram shown is a block diagram of an electronic device according to an embodiment of the multi-model vision control method of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0138] like Figure 7As shown, device 700 includes a computing unit 701, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 702 or a computer program loaded from storage unit 708 into random access memory (RAM) 703. RAM 703 may also store various programs and data required for the operation of device 700. The computing unit 701, ROM 702, and RAM 703 are interconnected via bus 704. Input / output (I / O) interface 705 is also connected to bus 704.

[0139] Multiple components in device 700 are connected to I / O interface 705, including: input unit 706, such as keyboard, mouse, etc.; output unit 707, such as various types of displays, speakers, etc.; storage unit 708, such as disk, optical disk, etc.; and communication unit 709, such as network card, modem, wireless transceiver, etc. Communication unit 709 allows device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0140] The computing unit 701 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as multi-model-based vision control methods. For example, in some embodiments, the multi-model-based vision control method can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 708.

[0141] In some embodiments, part or all of the computer program may be loaded and / or installed on the device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by computing unit 701, one or more steps of the multi-model-based vision control method described above may be performed. Alternatively, in other embodiments, computing unit 701 may be configured to perform the multi-model-based vision control method by any other suitable means (e.g., by means of firmware).

[0142] Various implementations of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include: implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transferring data and instructions to the storage system, the at least one input device, and the at least one output device.

[0143] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable multi-model-based vision control device, such that when executed by the processor or controller, the program code enables the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0144] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0145] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for showing information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0146] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0147] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is established by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service ecosystem, addressing the shortcomings of traditional physical hosts and VPS (Virtual Private Server, or simply "VPS") services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.

[0148] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0149] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A multi-model-based visual control method, comprising: Acquire the image to be processed and the corresponding target task information for the image to be processed; A first target model is obtained based on the target task information, and a task perception result is obtained based on the image to be processed using the first target model. A second target model is obtained based on the target task information. Using the second target model, a task inference result is obtained based on the image to be processed and the task perception result. In response to determining that the task reasoning result meets preset requirements, the target device is controlled to perform the target operation.

2. The method according to claim 1, wherein, The step of obtaining the first target model based on the target task information includes: Based on the target task information, determine the target perception object; Obtain the visual expert model corresponding to the target perception object, and use it as the first target model.

3. The method according to claim 1, further comprising: In response to determining that the first target model could not be obtained based on the target task information, the image to be processed and the target task information are sent to the server so as to use the multimodal large model located on the server to obtain the task inference result based on the image to be processed and the target task information; Receive the task reasoning results returned by the server.

4. The method according to claim 1, wherein, The step of obtaining the second target model based on the target task information includes: Based on the target task information, determine the target reasoning scenario; Obtain a contextualized visual large model corresponding to the target reasoning scenario, and use it as the second target model.

5. The method according to claim 1, further comprising: In response to determining that the task inference result could not be obtained based on the second target model, the image to be processed and the task perception result are sent to the server so that the multimodal large model located on the server can be used to obtain the task inference result based on the image to be processed and the task perception result. Receive the task reasoning results returned by the server.

6. The method according to claim 1, wherein, The step of controlling the target device to perform the target operation in response to determining that the task reasoning result meets the preset requirements includes: Based on the target task information, obtain the operation requirements and the target operations corresponding to the operation requirements; In response to determining that the task reasoning result meets the operational requirements, the target device is controlled to perform the target operation.

7. The method according to claim 1, further comprising: Obtain a first training set, which includes multiple first sample images and the perception annotation results of the corresponding target perception objects in the first sample images; The first sample image is input into the first neural network model to obtain the perceptual prediction result output by the first neural network model; The first loss function value is calculated based on the perception prediction results and perception annotation results of the plurality of first sample images. The parameters of the first neural network model are adjusted using the first loss function value to obtain the first target model.

8. The method according to claim 1, further comprising: Obtain a second training set, which includes multiple second sample images and the perception annotation results and inference annotation results of the target inference scene corresponding to the second sample images; The second sample image and its corresponding perceptual annotation results are input into the second neural network model to obtain the inference prediction results output by the second neural network model. The second loss function value is calculated based on the inference prediction results and inference annotation results of the multiple second sample images. The parameters of the second neural network model are adjusted using the second loss function value to obtain the second target model.

9. The method according to claim 3, further comprising: After sending the image to be processed and the target task information to the server, the system receives a first target model corresponding to the target task information returned by the server. The first target model is obtained by training a multimodal large model in the server using first labeled data corresponding to the target task information. The first target model is deployed locally on the target device.

10. A multi-model-based vision control device, comprising: An acquisition unit is used to acquire the image to be processed and the target task information corresponding to the image to be processed. A perception unit is used to obtain a first target model based on the target task information, and to obtain a task perception result based on the image to be processed using the first target model. The reasoning unit is used to obtain a second target model based on the target task information, and use the second target model to obtain a task reasoning result based on the image to be processed and the task perception result. The control unit is configured to control the target device to perform the target operation in response to determining that the task reasoning result meets the preset requirements.

11. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-9.

12. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-9.

13. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-9.

Images