Image processing method and device, processing equipment, storage medium and program product

Abstract

The invention discloses an image processing method and device, processing equipment, a storage medium and a program product. The method comprises the following steps: calling a first model to analyze a target image to obtain an analysis result; the target image is an image collected by a terminal; the analysis result comprises N sub-tasks required by execution of the target task, and N is a positive integer; for each sub-task in the N sub-tasks, calling a second model matched with the sub-task to process the sub-task to obtain a processing result; and sending the processing result of the N sub-tasks and / or operation information determined based on the processing result of the N sub-tasks to the terminal for presentation.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an image processing method and apparatus, processing equipment, storage medium, and program product. Background Art

[0002] With the development of artificial intelligence (AI), AI technology has been widely used in various industries. In operation and maintenance scenarios, images in real scenes can be processed based on AI models to achieve specific operation and maintenance tasks.

[0003] For specific operation and maintenance tasks, several subtasks need to be designed to implement them. For each subtask, the operation and maintenance personnel are required to take images of specific scenes and input the images of the specific scenes into the matching AI model for recognition. It can be seen that for different subtasks, corresponding interfaces need to be set for images of different scenes to call the matching AI model. If the image of a specific scene is input into an unmatched AI model, there will be a problem of unrecognition (that is, the corresponding subtask cannot be executed). In summary, the current solution for processing operation and maintenance tasks based on AI models has the problem of low efficiency. Summary of the invention

[0004] In order to solve the above technical problems, the present application provides an image processing method and apparatus, a processing device, a computer storage medium, and a computer program product.

[0005] The image processing method provided by this application includes:

[0006] Calling the first model to analyze the target image and obtain an analysis result; the target image is an image captured by the terminal; the analysis result includes N subtasks required for the execution of the target task, where N is a positive integer;

[0007] For each of the N subtasks, calling a second model matching the subtask to process the subtask and obtain a processing result;

[0008] The processing results of the N subtasks and / or the operation information determined based on the processing results of the N subtasks are sent to the terminal for presentation.

[0009] The image processing method provided by this application includes:

[0010] The collected image is sent to the server; the image is used by the server to obtain the processing results of N subtasks corresponding to the target task by calling the model, where N is a positive integer;

[0011] Receiving the processing results of the N subtasks and / or operation information determined based on the processing results of the N subtasks sent by the server;

[0012] A first virtual screen is displayed superimposed on the real scene, and the processing result and / or the operation information is presented on the first virtual screen.

[0013] The image processing device provided by the present application includes:

[0014] A first calling unit is used to call the first model to analyze a target image and obtain an analysis result; the target image is an image captured by the terminal; the analysis result includes N subtasks required for the execution of the target task, where N is a positive integer;

[0015] A second calling unit is used for calling a second model matching the subtask to process each of the N subtasks to obtain a processing result;

[0016] The first communication unit is used to send the processing results of the N subtasks and / or the operation information determined based on the processing results of the N subtasks to the terminal for presentation.

[0017] The image processing device provided by the present application includes:

[0018] An acquisition unit, used for acquiring images;

[0019] A second communication unit is used to send the image to a server; the image is used by the server to obtain the processing results of N subtasks corresponding to the target task by calling the model, where N is a positive integer; and receive the processing results of the N subtasks sent by the server and / or operation information determined based on the processing results of the N subtasks;

[0020] The display unit is used to display a first virtual screen superimposed on a real scene, wherein the processing result and / or the operation information is presented on the first virtual screen.

[0021] The processing device provided in the present application includes: a processor and a memory, the memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute any one of the above-mentioned image processing methods.

[0022] The computer-readable storage medium provided in the present application is used to store a computer program, and the computer program enables a computer to execute any one of the above-mentioned image processing methods.

[0023] The computer program product provided in the present application includes computer program instructions, and the computer program instructions enable a computer to execute any one of the above-mentioned image processing methods.

[0024] In the technical solution of the present application, for the image captured by the terminal, the first model is called to analyze and obtain the N subtasks required for the execution of the target task; for each subtask, the second model matching the subtask is adaptively called for processing; the processing results of the subtask and / or the operation information determined based on the processing results of the subtask are presented to the terminal through augmented reality (AR). It can be seen that the technical solution of the embodiment of the present application does not need to set corresponding interfaces for images of different scenes to call matching AI models. For images of real scenes, no matter what kind of scene the image is, it can be analyzed by the first model to obtain the N subtasks required for the execution of the target task. Further, for each subtask, the second model matching the subtask is adaptively called for processing, thereby improving the efficiency of processing the target task based on the AI ​​model. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1-1 It is a schematic diagram of the overall architecture provided by the embodiment of the present application;

[0026] Figure 1-2 It is a schematic diagram of the functional layers of the overall architecture provided by the embodiment of the present application;

[0027] Figure 2 1 is a flowchart of an image processing method provided in an embodiment of the present application;

[0028] Figure 3 is a schematic diagram of a process of analyzing a target image using a first model provided in an embodiment of the present application;

[0029] Figure 4 It is a schematic diagram of the process of image scene detection provided by an embodiment of the present application;

[0030] Figure 5 It is a schematic diagram of the key frame extraction process provided by the embodiment of the present application;

[0031] Figure 6 This is a schematic diagram of the image processing method provided in the embodiment of the present application. Figure 2 ;

[0032] Figure 7 Schematic diagram 1 of the interface presented by the AR glasses provided in an embodiment of the present application;

[0033] Figure 8 This is a schematic diagram of the interface presented by the AR glasses provided in the embodiment of the present application. Figure 2 ;

[0034] Fig. 9 This is a schematic diagram of the interface presented by the AR glasses provided in the embodiment of the present application Figure 3 ;

[0035] Fig.10 This is a schematic diagram of the interface presented by the AR glasses provided in the embodiment of the present application Figure 4 ;

[0036] Fig.11 It is a flow chart of application example 1 provided in the embodiment of the present application;

[0037] Fig.12 It is a flow chart of application example 2 provided in the embodiment of the present application;

[0038] Fig.13 It is a flow chart of application example 3 provided in the embodiment of the present application;

[0039] Fig.14 Schematic diagram 1 of the structure of the image processing device provided in the embodiment of the present application;

[0040] Fig.15 The structure of the image processing device provided in the embodiment of the present application is shown in FIG. Figure 2 ;

[0041] Fig.16 is a schematic structural diagram of a processing device provided in an embodiment of the present application;

[0042] Fig.17 It is a schematic structural diagram of the chip of an embodiment of the present application. DETAILED DESCRIPTION

[0043] The following will describe the technical solutions in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0044] Figure 1-1 This is the overall architecture diagram provided by the embodiment of the present application, such as Figure 1-1 As shown, the overall architecture provided by the embodiment of the present application includes: cloud side, edge side and end side. Among them, the cloud side is mainly responsible for cloud computing; the edge side is mainly responsible for edge computing; the end side is mainly responsible for data collection and interface display. The device corresponding to the end side is a terminal, such as: a smart phone, an AR device (such as smart glasses / AR glasses), etc. The device corresponding to the edge side is a server, which can be called an edge cloud server. The device corresponding to the cloud side is a server, which can be called a central cloud server. The edge side is located between the end side and the cloud side, and the corresponding interaction method is "end-edge-cloud" interaction. In some cases, the edge side can be omitted, and the corresponding interaction method is "end-cloud" interaction.

[0045] based on Figure 1-1 The overall architecture shown in the figure, the embodiment of the present application provides an image processing method based on model collaboration. The model collaboration here refers to the collaboration between large and small models (i.e., collaboration between a large model and a small model).

[0046] The large model is deployed on the cloud side, and this article refers to the large model as the first model. Figure 1-1 As shown in the figure, the following large models are deployed on the cloud side: a large visual model and a large language model.

[0047] The small model is deployed on the cloud side and / or the terminal side, and this article refers to the small model as the second model. Figure 1-1 As shown in the figure, the following small models are deployed on the edge: optical character recognition (OCR) recognition model, port recognition model, safety inspection model, equipment recognition model, face recognition model, process quality inspection model, hidden danger recognition model, and clothing quality inspection model. The following small models are deployed on the end: target recognition model, target tracking model, target classification model, service safety quality inspection model, optical cross port recognition model, clothing standard quality inspection model, equipment resource acceptance model, fiber jumper quality inspection model, equipment status recognition model, process quality inspection acceptance model, construction standard quality inspection model, and rack resource inventory model. Among them, the small model on the end side is more suitable for processing real-time data and has the characteristics of low processing latency; the small model on the edge side is more suitable for processing non-real-time data and has the characteristics of high processing latency.

[0048] It should be noted that different small models support / have different AI capabilities, and small models that support / have different AI capabilities can handle different subtasks. For example: the port identification model supports / has the port identification capability and can handle the port identification task; the dress code quality inspection model supports / has the dress code quality inspection capability and can handle the dress code quality inspection task.

[0049] It should be noted that the cloud side and / or the end side has a capability calling interface / model calling interface (referred to as the calling interface for short), which is used to call the small model on the cloud side and / or the end side.

[0050] It should be noted that the above Figure 1-1 The small models shown are only exemplary. In the operation and maintenance scenario, the types of small models involved can be more Figure 1-1 More or less.

[0051] It should be noted that the model described in the embodiment of the present application refers to a neural network model, which can also be called an AI model.

[0052] Figure 1-2 is a schematic diagram of the functional layers of the overall architecture provided by the embodiment of the present application, such as Figure 1-2As shown, the end side has the following functional layers: collection layer, business layer, and presentation layer; the cloud side has the following functional layers: large model layer; and the edge side has the following functional layers: small model layer. Among them, the collection layer of the end side includes a camera for collecting images / videos. The business layer of the end side includes inspection services, asset inventory services, resource inventory services, resource image recognition services, resource information acquisition services, etc. The presentation layer of the end side includes an AR module for superimposing and displaying virtual information in real scenes. The large model layer of the cloud side includes a large model (i.e., the first model). The small model layer of the edge side includes a small model (i.e., the second model). In addition, the end side, cloud side, and edge side also have an interface layer, through which the interaction between the end side, cloud side, and edge side is realized.

[0053] Figure 2 1 is a flow chart of an image processing method provided in an embodiment of the present application. The image processing method provided in an embodiment of the present application is applied to the server side. The server can also be understood as a backend device relative to the terminal, and the terminal can also be understood as a frontend device relative to the server. Figure 2 As shown, the image processing method provided in the embodiment of the present application includes some or all of the following contents:

[0054] Step 201: calling the first model to analyze the target image and obtain the analysis result; the target image is the image captured by the terminal; the analysis result includes N subtasks required for the execution of the target task, where N is a positive integer.

[0055] In the embodiment of the present application, the target image is an image of a real scene captured by the terminal. The terminal has a camera, and the terminal captures the image of the real scene through the camera.

[0056] In the embodiment of the present application, the terminal collects images of real scenes, which can be achieved by:

[0057] 1) The terminal collects video of the real scene in real time through the camera (i.e., recording), and the single frame image in the video can be understood as the target image mentioned in the embodiment of the present application.

[0058] 2) The terminal collects an image of the real scene through a camera (i.e., takes a photo), and the image can be understood as the target image mentioned in the embodiments of the present application.

[0059] In the embodiments of the present application, the real scene refers to a real scene, such as a computer room, a construction site, etc.

[0060] Exemplarily, a series of devices are deployed in the real scene, such as: servers, network devices (such as switches, routers, etc.), storage devices (such as hard disk arrays, tape libraries, etc.), security devices (such as firewalls), communication cables, etc. The terminal captures an image in the real scene from a specific perspective, and one or more devices are presented in the image.

[0061] In an embodiment of the present application, the terminal sends the captured image of the real scene (i.e., the target image) to the server; after the server receives the target image, it processes the target image in a model collaborative manner.

[0062] The model collaboration in the embodiments of the present application refers to the collaboration of large and small models (i.e., collaboration between large and small models). Among them, the large model is deployed on the cloud side, and the large model is referred to as the first model in this article. The small model is deployed on the end side and / or the edge side, and the small model is referred to as the second model in this article. The first model is a global model, and the second model is a local model; the global model refers to a model that has the ability to call the local model; different local models have different AI capabilities.

[0063] In some embodiments, the first model includes a large video model and a large language model (Large Language Model, LLM), and the first model is used to analyze the target image to obtain an analysis result. Here, the analysis result obtained by the large model includes N subtasks required for the execution of the target task, where N is a positive integer. Here, the N subtasks can also be understood as N steps.

[0064] In some implementations, the target task includes at least one of the following: a machine room inspection task, a construction site survey task, an equipment acceptance task, and an equipment maintenance task.

[0065] Exemplarily, a target device (such as a server, a network device, a storage device, a communication cable, etc.) is presented in a target image, and the target image is input into the first model. By analyzing the target image through the first model, N steps for accepting the target device can be obtained, wherein "accepting the target device" is also "target task", and "N steps" are also "N subtasks". Taking N=2 as an example, accepting the target device requires steps S11 and S12, wherein step S11 is for identifying basic information of the target device (such as the type of the target device, the model of the target device, etc.); step S12 is for identifying the status of the target device (such as the cable connection status of the target device, the working status of the target device, etc.).

[0066] For each of the N subtasks, a second model matching the subtask may be called for processing, as detailed in the following steps.

[0067] Step 202: For each of the N subtasks, call the second model matching the subtask to process the subtask and obtain a processing result.

[0068] Here, the second model matching the subtask refers to a local model having AI capabilities for processing the subtask.

[0069] In some embodiments, the second model includes one or more of the following: OCR recognition model, port recognition model, safety inspection model, equipment recognition model, face recognition model, process quality inspection model, hidden danger recognition model, clothing quality inspection model, target recognition model, target tracking model, target classification model, service safety quality inspection model, optical cross port recognition model, dress code quality inspection model, equipment resource acceptance model, fiber jumper quality inspection model, equipment status recognition model, process quality inspection acceptance model, construction code quality inspection model, rack resource inventory model. In the operation and maintenance scenario, the types of small models involved can be more or less than the above.

[0070] In the embodiment of the present application, different second models may have different AI capabilities, or in other words, different second models can process different subtasks. For example, the following Table 1 lists several models and their corresponding AI capabilities.

[0071]

[0072]

[0073] Table 1: AI capabilities corresponding to the models

[0074] For each of the N subtasks determined in step 201, a second model matching the subtask is called to process the subtask. Here, the second model matching the subtask refers to a model with AI capabilities for processing the subtask.

[0075] Exemplarily, taking N=2 as an example, the two subtasks correspond to step S11 and step S12, respectively, wherein step S11 is to identify the basic information of the target device (such as the type of the target device, the model of the target device, etc.); step S12 is to identify the state of the target device (such as the cable connection state of the target device, the working state of the target device, etc.). For these two subtasks, according to Table 1, the device identification model is called to process the first subtask, i.e., to identify the basic information of the target device, and then the device state identification model is called to process the second subtask, i.e., to identify the state of the target device.

[0076] In some implementations, if there are multiple second models that match the subtask, a second model is selected from the multiple matching second models based on the model deployment location to process the subtask.

[0077] Exemplarily, the server searches for a model that matches the subtask among all models. If multiple models are found to match the subtask, the server selects a model whose deployment location is closest to the terminal from the multiple matching models to process the subtask. By selecting a model whose deployment location is closest to the terminal to process the subtask, the processing delay can be effectively reduced. It should be noted that the models mentioned here all refer to the second model.

[0078] Step 203: Send the processing results of the N subtasks and / or the operation information determined based on the processing results of the N subtasks to the terminal for presentation.

[0079] In some implementations, the processing results of the N subtasks are sent to the terminal for presentation, so that the user on the terminal side can clearly understand the processing result of the target task.

[0080] Exemplarily, taking "acceptance of target device" as the target task, the two subtasks (i.e., the case where N=2) correspond to step S11 and step S12, respectively, wherein step S11 is to identify the basic information of the target device (such as the type of target device, the model of the target device, etc.); step S12 is to identify the status of the target device (such as the cable connection status of the target device, the working status of the target device, etc.). For these two subtasks, the device identification model is called to process the first subtask, i.e., to identify the basic information of the target device, and then the device status identification model is called to process the second subtask, i.e., to identify the status of the target device. The processing result of the first subtask is: the target device is a switch and the model is a type A switch; the processing result of the second subtask is: the cable connection of the target device is correct and works normally; the user on the terminal side can clearly know that the acceptance of the target device is successful based on the processing results of these two subtasks.

[0081] In some implementations, operation information is determined based on the processing results of the N subtasks, and the operation information is sent to the terminal for presentation, so that the user at the terminal side can clearly understand the operation taken on the target device.

[0082] Exemplarily, taking "maintaining the target device" as the target task, the two subtasks (i.e., the case where N=2) correspond to step S11 and step S12, respectively, wherein step S11 is to identify the basic information of the target device (such as the type of the target device, the model of the target device, etc.); step S12 is to identify the state of the target device (such as the cable connection state of the target device, the working state of the target device, etc.). For these two subtasks, the device identification model is called to process the first subtask, i.e., to identify the basic information of the target device, and then the device state identification model is called to process the second subtask, i.e., to identify the state of the target device. The processing result of the first subtask is: the target device is a switch and the model is a type A switch; the processing result of the second subtask is: the cable connection of the target device is wrong and does not work. The server side generates operation information based on the processing results of the two subtasks, and the operation information instructs the user on the terminal side to repair the cable connection method of the target device. The user on the terminal side repairs the cable connection method of the target device according to the prompt of the operation information, thereby achieving the purpose of equipment maintenance.

[0083] In some embodiments, the processing results of the N subtasks and / or the operation information determined based on the processing results of the N subtasks are sent to the remote terminal corresponding to the terminal for presentation. Here, both the terminal and the remote terminal will present the processing results of the N subtasks and / or the operation information determined based on the processing results of the N subtasks, so that it is convenient for the remote terminal to synchronously obtain the processing results and / or operation information, and the user on the remote terminal side can give some relevant guidance or suggestions to the user on the terminal side based on the processing results and / or operation information, for example, the user on the remote terminal side provides maintenance means for the user on the terminal side.

[0084] In some implementations, training data is generated based on the target image and the processing results of the N subtasks, and the first model is trained using the training data. In this way, the first model can be continuously trained and optimized during use.

[0085] It should be noted that the target object / target device described in the embodiments of the present application may be, but is not limited to, a dumb device, which may also be referred to as a dumb resource, and is a passive device.

[0086] The technical solution of the embodiment of the present application proposes an image processing method based on the collaboration of large and small models. Through the collaboration of large and small models, a large number of images can be quickly and accurately analyzed and processed, thereby improving operation and maintenance efficiency.

[0087] Figure 3 2 is a schematic diagram of a process of analyzing a target image using a first model according to an embodiment of the present application. The process of analyzing a target image using a first model according to an embodiment of the present application can be implemented in combination with step 201 described above; Figure 3 As shown, the process of analyzing the target image by the first model provided in the embodiment of the present application includes part or all of the following contents (it should be noted that the following steps are related steps for the first model to analyze the target image):

[0088] Step 301: Identify key information of a target image, and determine scene information corresponding to the target image according to the key information.

[0089] In the embodiment of the present application, a target object is presented in the target image, and the scene where the target object is located is the scene corresponding to the target image described in this article. The description of "target object" can also be replaced by "target device". Exemplarily, the target object is a server, a network device (such as a switch, a router, etc.), a storage device (such as a hard disk array, a tape library, etc.), a security device (such as a firewall), a communication cable, etc.

[0090] In some embodiments, the key information of the target image includes, but is not limited to: information about the target objects in the target image (such as the number of target objects, the appearance of target objects, the type of target objects, the behavior of target objects, etc.), information about the environment in the target image (such as the type of environment, characteristics of the environment, etc.).

[0091] In the embodiment of the present application, there is an association relationship between the key information and the scene information, and the scene information corresponding to the key information can be determined based on the association relationship.

[0092] In some implementations, the scene information (such as scene type) corresponding to the target image is used to identify the scene corresponding to the target image. The scenes corresponding to the target image include but are not limited to: device identification scenes, business association scenes, environment status identification scenes, tag identification scenes, etc.

[0093] Step 302: Determine N audit points associated with the target image based on scene information corresponding to the target image, and the N audit points have a corresponding relationship with the N subtasks.

[0094] In the embodiment of the present application, the network operation and maintenance domain knowledge base (referred to as the network domain knowledge base) is used to determine the audit point (also described as an identification item) in combination with the scene information corresponding to the target image. Here, the network domain knowledge base records the association between the scene information and the audit point.

[0095] For example, for a device identification scenario, the audit points associated with the scenario include: identifying the device type in the image, where the device type includes, for example, a switch, a router, a firewall, a server, and the like.

[0096] Exemplarily, for a business-related scenario, the audit points associated with the scenario include: analyzing the business direction, connection status, interface status, etc. of the optical cable.

[0097] Exemplarily, for the environmental status identification scenario, the audit points associated with the scenario include: checking the following inspection indicators specified in the computer room: sanitary tools, air conditioning, wiring, and blocking.

[0098] Exemplarily, for the label identification scenario, the associated audit points include: confirming whether the device label is clear and identifying the label information.

[0099] It should be noted that one scene information can be associated with one or more audit points.

[0100] In an embodiment of the present application, a network domain knowledge base is formed by performing scene analysis on a large number of images and marking the audit points corresponding to the images. In specific implementation, the potential abnormalities of different scenarios are determined according to the actual network operation and maintenance situation, and these potential abnormalities are used as audit points associated with the scenario. Exemplarily, the audit points include but are not limited to: identifying fault points (such as identifying whether the cable is broken, identifying whether the equipment indicator light is abnormal, identifying whether the inspection indicators are inconsistent, etc.), analyzing safety hazards (such as analyzing whether the cables are messy, analyzing whether the equipment is operating improperly, etc.). A network domain knowledge base is constructed or maintained based on the audit points associated with different scenarios. Furthermore, for the target image, the scene information of the target image is compared with the entries in the network domain knowledge base (the entries record the scene information and the corresponding audit points), and the entries that match the scene information are found, and the audit points in the matching entries are used as the audit points of the target image.

[0101] Step 303: Perform intent recognition based on the scene information corresponding to the target image and the N audit points associated with the target image to obtain an intent recognition result; the intent recognition result represents the target task.

[0102] In the embodiment of the present application, intent recognition is intended to determine the intent (or purpose) expressed by the "input". In other words, intent recognition is to understand the "input" and analyze the target task to be performed. The "input" here includes the scene information corresponding to the target image and the N audit points associated with the target image.

[0103] In some embodiments, the "input" of intent recognition also includes user instructions; specifically, obtaining user instructions collected by the terminal; determining the user intent based on the user instructions, and performing intent recognition based on the user intent, scene information corresponding to the target image, and N audit points associated with the target image, thereby improving the accuracy of intent recognition.

[0104] In a specific implementation, when a user instruction is obtained, the user instruction is subjected to lexical analysis, syntactic analysis and semantic analysis through the first model, so as to understand the user intention.

[0105] Step 304: Determine N subtasks to complete the target task based on the intention recognition result.

[0106] In the embodiment of the present application, after the target task is determined, the target task is disassembled to obtain N subtasks for completing the target task.

[0107] In the embodiment of the present application, the first model (i.e., the large model) is usually a heavyweight model, which is characterized by a complex model structure and a large number of model parameters, etc.; the first model is suitable for processing complex and global tasks, which correspond to the above steps 301 to 304. Correspondingly, the second model (i.e., the small model) is usually a lightweight model, which is characterized by a simple model structure and a small number of model parameters, etc.; the second model is suitable for processing simple and local tasks.

[0108] Figure 4 3 is a flow chart of the image scene detection provided in the embodiment of the present application. The process of the image scene detection provided in the embodiment of the present application can be implemented in combination with step 301 described above; Figure 4 As shown, the process of image scene detection provided by the embodiment of the present application includes some or all of the following contents:

[0109] Step 401: Identify key information of the target image.

[0110] In the embodiment of the present application, the key information of the target image can be identified by the following process:

[0111] 1) Extract the feature vector of the target image.

[0112] Here, the feature vector of the target image represents the semantic features of the target image.

[0113] In some embodiments, a pre-trained image recognition model (such as a model based on a residual network (ResNet) / Visual Geometry Group (VGG)) is used to extract semantic features (such as high-level semantic features) of a target image to obtain a feature vector of the target image.

[0114] 2) Convert the feature vector of the target image into text.

[0115] Here, the purpose of converting the feature vector of the target image into text is to describe the target image through text.

[0116] In some embodiments, an image description generation model (such as a model based on a convolutional neural network (CNN) / recurrent neural network (RNN)) is used to convert a feature vector of a target image into text, where the text is a description of the content presented in the target image and includes key information in the target image, such as information about target objects in the target image (such as the number of target objects, the appearance of target objects, the type of target objects, the behavior of target objects, etc.) and information about the environment in the target image (such as the type of environment, characteristics of the environment, etc.).

[0117] It should be noted that the purpose of the above-mentioned “extracting the feature vector of the target image” and “converting the feature vector of the target image into text” is to convert the image data of the target image into a format that can be understood by the large language model.

[0118] It should be noted that the large video model may include the above-mentioned image recognition model and image description generation model.

[0119] 3) Extract key information from the text.

[0120] Here, a large language model is used to extract key information from the text. Here, the large language model can understand and analyze the text and identify key information from the text, such as information about the target objects in the target image (such as the number of target objects, the appearance of the target objects, the type of the target objects, the behavior of the target objects, etc.), and information about the environment in the target image (such as the type of the environment, the characteristics of the environment, etc.).

[0121] Step 402: Determine scene information corresponding to the target image according to the key information.

[0122] Here, the large language model is used to determine the scene information corresponding to the target image according to the key information. In the embodiment of the present application, the key information and the scene information have an association relationship, and the scene information corresponding to the key information can be determined according to the association relationship.

[0123] In addition, after extracting the feature vector of the target image and converting the feature vector of the target image into text, the large language model can also directly understand the semantic information of the text, thereby inferring the scene information corresponding to the target image.

[0124] In some implementations, the scene information (such as scene type) corresponding to the target image is used to identify the scene corresponding to the target image. The scenes corresponding to the target image include but are not limited to: device identification scenes, business association scenes, environment status identification scenes, tag identification scenes, etc.

[0125] In some embodiments, in order to more accurately achieve image scene detection, the embodiment of the present application adopts a multimodal learning method. Specifically, the target image and text are simultaneously input into the multimodal model, and the target image and text are simultaneously processed by the multimodal model to obtain more accurate scene information. Here, the multimodal model includes a large video model and a large language model; wherein the large video model is used to process the target image to obtain the corresponding first scene information; the large language model is used to process the text to obtain the corresponding second scene information; and the final scene information is determined based on the first scene information and the second scene information.

[0126] Figure 5 is a schematic diagram of the key frame extraction process provided by the embodiment of the present application. The key frame extraction process provided by the embodiment of the present application can be combined with the key frame extraction process described above. Figure 2 The implementation sequence may be, but is not limited to, before step 201; Figure 5 As shown, the key frame extraction process provided in the embodiment of the present application includes some or all of the following contents:

[0127] Step 501: Obtain the video of the real scene captured by the terminal.

[0128] Step 502: Detect key areas and key time periods in the video.

[0129] In the embodiment of the present application, the spatiotemporal region in the video is detected by the video time sequence action detection (R-C3D) model, and the spatiotemporal region includes two dimensions, one dimension is the key time period (corresponding to the time domain), and the other dimension is the key area (corresponding to the air domain). Among them, the key time period refers to the time period in which the target object and / or target action exists in the video, and the key area refers to the area in which the target object and / or target action exists in the video. The key area and key time period are the basis for extracting key frames.

[0130] In the embodiment of the present application, the R-C3D model can be trained by a video sample set, thereby increasing the generalization ability of the R-C3D model. The video sample set here can be constructed by, but is not limited to, the following methods: 1) fast-forwarding a standard video to form a video sample; 2) slowing down a standard video to form a video sample; 3) inserting a frame into a standard video to form a video sample.

[0131] In some embodiments, in order to more accurately detect key areas and key time periods in a video, low-level visual features (such as color, texture, etc.) and high-level semantic features (such as the category of the target object, the pattern of the target action, etc.) of each frame image in the video are pre-extracted, and the low-level visual features and high-level semantic features are fused into a multi-level feature representation. The multi-level feature representation is processed by the R-C3D model to enhance the R-C3D model's ability to understand the video content, thereby more accurately detecting key areas and key time periods in the video.

[0132] In some embodiments, in order to more accurately detect key areas and key time periods in a video, each frame of the video is optimized in advance. Specifically, a dynamic contrast adjustment algorithm is used to adjust the contrast of each frame of the video to enhance the visual significance of the key areas and make the extraction of key frames more accurate.

[0133] Step 503: extract key frames from the video based on the key areas and key time periods, and use the key frames as target images.

[0134] Here, a key frame refers to a frame image in a video that contains a key area and / or is located in a key time period.

[0135] In some embodiments, a series of key frames extracted from a video based on key areas and key time periods form a key frame candidate set; each key frame in the key frame candidate set is sorted by a comprehensive evaluation model; a key frame selection threshold N is determined according to the complexity and dynamic change state of the video content, and the top N key frames are selected from the key frame candidate set according to the selection threshold, and the N key frames are all used as target images. The comprehensive evaluation model here can be constructed based on the visual features, semantic features, and spatiotemporal features corresponding to the frame image.

[0136] The technical solution of the embodiment of the present application extracts key frames through the R-C3D model, and processes the key frames as target images, thereby avoiding processing invalid frames in the video, improving processing efficiency, and saving processing resources.

[0137] Figure 6 This is a schematic diagram of the image processing method provided in the embodiment of the present application. Figure 2 The image processing method provided in the embodiment of the present application is applied to a terminal. The terminal can be understood as a front-end device relative to the server, and the server can be understood as a back-end device relative to the terminal; Figure 6 As shown, the image processing method provided in the embodiment of the present application includes some or all of the following contents:

[0138] Step 601: Capture an image and send the image to a server; the image is used by the server to obtain the processing results of N subtasks corresponding to the target task by calling a model, where N is a positive integer.

[0139] In the embodiment of the present application, the terminal collects images of the real scene. The terminal has a camera, and the terminal collects images of the real scene through the camera.

[0140] In the embodiment of the present application, the terminal collects images of real scenes, which can be achieved by:

[0141] 1) The terminal collects video of the real scene in real time through the camera (i.e., recording), and the single frame image in the video can be understood as the target image mentioned in the embodiment of the present application.

[0142] 2) The terminal collects an image of the real scene through a camera (i.e., takes a photo), and the image can be understood as the target image mentioned in the embodiments of the present application.

[0143] In the embodiments of the present application, the real scene refers to a real scene, such as a computer room, a construction site, etc.

[0144] Exemplarily, a series of devices are deployed in the real scene, such as: servers, network devices (such as switches, routers, etc.), storage devices (such as hard disk arrays, tape libraries, etc.), security devices (such as firewalls), communication cables, etc. The terminal captures an image in the real scene from a specific perspective, and one or more devices are presented in the image.

[0145] In the embodiment of the present application, the terminal sends the captured image of the real scene (i.e., the target image) to the server; after receiving the target image, the server processes the target image in a model collaborative manner. Here, the implementation method of "the server processes the target image in a model collaborative manner" can be referred to in the above Figure 2 Related description.

[0146] Step 602: receiving the processing results of N subtasks and / or operation information determined based on the processing results of the N subtasks sent by the server; superimposing and displaying a first virtual screen in the real scene, wherein the processing results and / or operation information are presented on the first virtual screen.

[0147] The terminal in the embodiment of the present application is an AR device (such as AR glasses), and the terminal has a display device, through which a first virtual screen can be superimposed and displayed in a real scene, and the processing results and / or operation information are presented on the first virtual screen.

[0148] The processing result in the embodiment of the present application is associated with a target object in a real scene, and illustratively, the target object is a server, a network device (such as a switch, a router, etc.), a storage device (such as a hard disk array, a tape library, etc.), a security device (such as a firewall), a communication cable, etc. The first virtual screen is superimposed and displayed in the real scene, specifically including: displaying a label at a first position relative to the target object in the real scene through a virtual display effect, the label is used to indicate the target object; and displaying the processing result and / or operation information at a second position relative to the label through a virtual display effect.

[0149] Here, the display form of the annotation may include but is not limited to: arrow, box, highlight area. The purpose of displaying the annotation is to guide the user to pay attention to the target object. The first position relative to the target object may be the position where the target object is located or a position around the target object.

[0150] Here, the display form of the processing result and / or operation information may include but is not limited to: text, picture. The purpose of displaying the processing result and / or operation information is to inform the user of the processing result and / or operation information for the target object. The second position relative to the annotation may be the position where the annotation is located or a position around the annotation.

[0151] In some implementations, the terminal sends an image (such as an image of a real scene) to the remote terminal; the image is used by the remote user to provide operation suggestions on the remote terminal; the operation suggestions sent by the remote terminal are received; a second virtual screen is superimposed on the real scene, and the operation suggestions are presented on the second virtual screen. In this way, remote guidance can be achieved between the terminal and the remote terminal; for remote users, they can deeply experience the real scene without going to the scene, so as to find problems in the real scene, and provide operation suggestions to the on-site users through remote guidance, so as to properly handle the problems.

[0152] In some embodiments, the terminal identifies information about a target object in a real scene based on an image (such as an image of a real scene); and displays a third virtual screen superimposed on the real scene, on which information about the target object is presented. Here, the information about the target object includes, but is not limited to, the number of target objects, the appearance of the target objects, the type of the target objects, the behavior of the target objects, and other information. In this way, on-site users can obtain information about the target objects in real time, understand the situation of the target objects, and promptly discover whether there are any problems with the target objects.

[0153] In some embodiments, the terminal receives a task list sent by the server, the task list includes one or more target tasks; a fourth virtual screen is superimposed and displayed in the real scene, and one or more of the following contents are presented on the fourth virtual screen: the task list, the target tasks that have been executed in the task list, and the target tasks that have not been executed in the task list. In this way, the on-site user can standardize the execution order of tasks according to the task list to avoid executing tasks in a disorderly manner.

[0154] In one example, if Figure 7 As shown, Figure 7 It is an interface presented by AR glasses. When the user wears AR glasses during the inspection process, the AR glasses collect images in the real scene, identify the information of the target device in the real scene by analyzing the image, and superimpose the information of the target device in the real scene in a virtual way. Here, the information of the target device includes but is not limited to: device type, device model, device number, device status, maintenance record, etc. As an implementation method, a QR code is attached to the target device. After the AR glasses collect the image, the information of the target device can be obtained by identifying the QR code in the image. As another implementation method, after the AR glasses collect the image, the information of the target device can be obtained by identifying the appearance of the target device in the image. In this way, the user can obtain the information of the target device without viewing paper documents or electronic screens, which assists the user in completing the inspection task.

[0155] In one example, if Figure 8 As shown, Figure 8 It is another interface presented by AR glasses. When the user wears AR glasses during the inspection process, AR glasses receive the task list sent by the server, and superimpose the task list, the executed target tasks in the task list, and the unexecuted target tasks in the task list on the real scene in a virtual way, so as to avoid omissions during the inspection process. As an implementation method, for each task, the user can interact with the AR glasses (for example, through gestures, voice, etc.). For example, the user instructs the AR glasses to display the task details page, and the AR glasses superimpose the task details page on the real scene in a virtual way to assist the user in completing the task. Here, the task details page includes but is not limited to: the task name, the location corresponding to the task, and the indicator corresponding to the task.

[0156] In one example, if Fig. 9 As shown, Fig. 9It is another interface presented by AR glasses. When users wear AR glasses during the inspection process, AR glasses collect images in the real scene and transmit the images to the server in real time. The server processes the images through the collaboration of large and small models and returns the processing results to the AR glasses. Here, taking "the processing result is a cable connection error of the target device" as an example, AR glasses superimpose and display annotations in the problem area (such as the cable area of ​​the target device) in a virtual way. The annotations can be virtual arrows to directly guide users to pay attention to the problem area and improve the accuracy and efficiency of the inspection. AR glasses also superimpose the processing results (such as cable connection errors) in the real scene in a virtual way around the annotations. Furthermore, the server also returns the operation information to the AR glasses. Here, taking "the operation information is the repair steps" as an example, AR glasses superimpose and display the operation information (such as the repair steps) in a virtual way around the annotations.

[0157] In one example, if Fig.10 As shown, Fig.10 It is another interface presented by AR glasses. When users wear AR glasses during the inspection process, AR glasses collect images in the real scene and transmit the images to the remote terminal in real time, so that remote users can view the real scene in real time from the first-person perspective. They can deeply experience the real scene without going to the site, thereby discovering problems in the real scene and providing operation suggestions to on-site users through remote guidance. AR glasses superimpose the operation suggestions sent by the remote terminal on the real scene to assist on-site users in completing inspection tasks. In this way, on-site users can conduct on-site inspections through AR glasses, and what on-site users see can be intuitively presented to remote users. Remote users can obtain the most realistic situation as if they were on site, creating an intelligent, digital, efficient, and low-cost operation and maintenance model.

[0158] In some embodiments, the images captured by the terminal and the virtual screens displayed by the terminal will be saved in the cloud to form a powerful video knowledge base. The data in the video knowledge base is used to optimize the AI ​​recognition algorithm (the AI ​​recognition algorithm recognizes images and displays recognition information through virtual screens) to improve the recognition accuracy and efficiency of the AI ​​recognition algorithm.

[0159] The technical solution of the embodiment of the present application utilizes AR technology to integrate virtual information into real scenes, providing users with an intuitive and convenient interaction method, making the operation and maintenance process more intuitive, and enabling edge operation and maintenance personnel to quickly get started, thereby improving the work efficiency of the operation and maintenance personnel.

[0160] The technical solution of the embodiment of the present application is illustrated below with reference to specific application examples.

[0161] Application Example 1

[0162] This application example uses the computer room inspection scenario as an example. Fig.11 As shown in the figure, in the equipment room inspection scenario, the following processes are included:

[0163] 1) Arrange inspection tasks for the computer room.

[0164] 2) Operation and maintenance personnel arrive at the computer room.

[0165] 3) The operation and maintenance personnel begin the inspection tasks, which include: computer room environment inspection, equipment information identification, rack information identification, and remote fault handling.

[0166] 4) Operation and maintenance personnel wear AR glasses to collect images of the computer room environment, equipment, racks, and fault areas, and process each collected image through the collaboration of large and small models.

[0167] 5) AR glasses display the processing results in a virtual way by superimposing them on the real scene.

[0168] Application Example 2

[0169] This application example uses the construction site survey scenario as an example. Fig.12 As shown in the figure, in the construction site survey scenario, the following processes are included:

[0170] 1) Arrange site survey tasks.

[0171] 2) Operation and maintenance personnel arrive at the construction site.

[0172] 3) Operation and maintenance personnel begin the survey task, which includes: construction safety inspection.

[0173] 4) The operation and maintenance personnel wear AR glasses to collect images of the construction site. The content of the image is a worker wearing a hard hat and driving an excavator. The image is processed by the collaboration of large and small models. Specifically, the subtasks corresponding to the construction safety inspection are identified by the large model, including: Task 1 (detecting whether the hard hat is worn); Task 2 (identifying whether the excavator has safety hazards); for Task 1, call small model 1 (i.e., hard hat detection model) to detect whether the worker in the image is wearing a hard hat; for Task 2, call small model 2 (i.e., excavator detection model) to identify whether the excavator in the image has safety hazards.

[0174] 5) AR glasses superimpose the processing results on the real scene in a virtual way. Specifically, the processing results include: the worker wears the helmet correctly and the excavator is working safely.

[0175] Application Example 3

[0176] This application example explains the process of large and small model collaboration. Fig.13As shown in the figure, the process of large and small model collaboration includes the following steps:

[0177] 1) Get the image captured by the terminal.

[0178] 2) Image scene detection: The process of image scene detection can refer to the above Figure 4 Related description.

[0179] 3) Audit point determination: The audit point process can refer to the above Figure 3 Description related to step 302 in .

[0180] 4) Intent recognition: The process of intent recognition can refer to the above Figure 3 Description related to step 303 in .

[0181] 5) Task decomposition: The process of task decomposition can refer to the above Figure 3 Description related to step 304 in .

[0182] 6) Calling the small model: The process of calling the small model can refer to the above Figure 2 Description related to step 202 in FIG.

[0183] 7) AR displays processing results and / or operation information: The implementation method of AR displays processing results and / or operation information can refer to the above Figure 6 Related description.

[0184] 8) The images collected by the terminal and the corresponding processing results and / or operation information are recorded as historical records, and the historical records are used to train and optimize the large model.

[0185] Fig.14 Schematic diagram of the structure of the image processing device provided in the embodiment of the present application, which is applied to a server, such as Fig.14 As shown, the image processing device includes:

[0186] The first calling unit 1401 is used to call the first model to analyze the target image and obtain an analysis result; the target image is an image captured by the terminal; the analysis result includes N subtasks required for the execution of the target task, where N is a positive integer;

[0187] The second calling unit 1402 is used for calling a second model matching the subtask to process each of the N subtasks and obtain a processing result;

[0188] The first communication unit 1403 is used to send the processing results of the N subtasks and / or the operation information determined based on the processing results of the N subtasks to the terminal for presentation.

[0189] In some implementations, the first calling unit 1401 is used to call the first model to perform the following analysis on the target image:

[0190] Identify key information of the target image, and determine scene information corresponding to the target image according to the key information;

[0191] Determining N audit points associated with the target image based on scene information corresponding to the target image, wherein the N audit points have a corresponding relationship with the N subtasks;

[0192] Performing intention recognition based on scene information corresponding to the target image and N audit points associated with the target image to obtain an intention recognition result; the intention recognition result represents the target task;

[0193] Determine N subtasks for completing the target task based on the intention recognition result.

[0194] In some embodiments, identifying the key information of the target image includes: extracting a feature vector of the target image and converting the feature vector of the target image into text; wherein the feature vector represents a semantic feature of the target image; and extracting the key information from the text.

[0195] In some embodiments, the performing of intent recognition based on the scene information corresponding to the target image and N audit points associated with the target image includes: obtaining user instructions collected by a terminal; determining the user intent based on the user instructions, and performing intent recognition based on the user intent, the scene information corresponding to the target image, and N audit points associated with the target image.

[0196] In some embodiments, the first model is a global model, and the second model is a local model; the global model refers to a model having the ability to call the local model; different local models have different artificial intelligence (AI) capabilities; wherein the second model matching the subtask refers to a local model having the AI ​​capability to process the subtask.

[0197] In some implementations, the target task includes at least one of the following: a machine room inspection task, a construction site survey task, an equipment acceptance task, and an equipment maintenance task.

[0198] In some implementations, the target image is an image of a real scene captured by a terminal.

[0199] In some embodiments, the image processing device also includes: a key frame extraction unit 1404, which is used to obtain a video of a real scene captured by a terminal; detect key areas and key time periods in the video; the key area refers to the area in the video where the target object and / or target action exists; the key time period refers to the time period in the video where the target object and / or target action exists; based on the key area and the key time period, key frames are extracted from the video, and the key frames are used as the target image.

[0200] In some implementations, the second calling unit 1402 is configured to select a second model from the plurality of matching second models to process the subtask based on a model deployment location if there are a plurality of second models matching the subtask.

[0201] In some embodiments, the device further includes: a training unit, configured to generate training data according to the target image and the processing results of the N subtasks, and train the first model using the training data.

[0202] In some implementations, the first communication unit 1403 is used to send the processing results of the N subtasks and / or operation information determined based on the processing results of the N subtasks to a remote terminal corresponding to the terminal for presentation.

[0203] Those skilled in the art should understand that Fig.14 The implementation functions of each unit in the image processing device shown can be understood by referring to the relevant description of the aforementioned method. Fig.14 The functions of each unit in the image processing device shown may be implemented by a program running on a processor, or may be implemented by a specific logic circuit.

[0204] Fig.15 The structure of the image processing device provided in the embodiment of the present application is shown in FIG. Figure 2 , applied to terminals, such as Fig.15 As shown, the image processing device includes:

[0205] The acquisition unit 1501 is used to acquire images;

[0206] The second communication unit 1502 is used to send the image to the server; the image is used by the server to obtain the processing results of N subtasks corresponding to the target task by calling the model, where N is a positive integer; receive the processing results of the N subtasks sent by the server and / or operation information determined based on the processing results of the N subtasks;

[0207] The display unit 1503 is used to display a first virtual screen superimposed on the real scene, and the first virtual screen presents the processing result and / or the operation information.

[0208] In some embodiments, the processing result is associated with a target object in the real scene; the display unit 1503 is used to display a label through a virtual display effect at a first position relative to the target object in the real scene, and the label is used to indicate the target object; and to display the processing result and / or the operation information through a virtual display effect at a second position relative to the label.

[0209] In some implementations, the second communication unit 1502 is used to send the image to a remote terminal; the image is used by a remote user to provide operation suggestions on the remote terminal; and receive the operation suggestions sent by the remote terminal;

[0210] The display unit 1503 is used to display a second virtual screen superimposed on the real scene, and the operation suggestion is presented on the second virtual screen.

[0211] In some embodiments, the image processing device further includes: a recognition unit, configured to recognize information of a target object in the real scene based on the image;

[0212] The display unit 1503 is used to display a third virtual picture superimposed on the real scene, and the third virtual picture presents the information of the target object.

[0213] In some implementations, the second communication unit 1502 is used to receive a task list sent by the server, where the task list includes one or more target tasks;

[0214] The display unit 1503 is used to display a fourth virtual screen superimposed on the real scene, and the fourth virtual screen presents one or more of the following contents: the task list, the target tasks that have been executed in the task list, and the target tasks that have not been executed in the task list.

[0215] In some embodiments, the image is an image of the real scene.

[0216] Those skilled in the art should understand that Fig.15 The implementation functions of each unit in the image processing device shown can be understood by referring to the relevant description of the aforementioned method. Fig.15 The functions of each unit in the image processing device shown may be implemented by a program running on a processor, or may be implemented by a specific logic circuit.

[0217] Fig.161600 is a schematic structural diagram of a processing device 1600 provided in an embodiment of the present application. The processing device may be a server or a terminal. Fig.16 The processing device 1600 shown includes a processor 1610, which can call and run a computer program from a memory to implement the method in the embodiment of the present application.

[0218] Alternatively, if Fig.16 As shown, the processing device 1600 may further include a memory 1620. The processor 1610 may call and run a computer program from the memory 1620 to implement the method in the embodiment of the present application.

[0219] The memory 1620 may be a separate device independent of the processor 1610 , or may be integrated into the processor 1610 .

[0220] Alternatively, if Fig.16 As shown, the processing device 1600 may further include a transceiver 1630, and the processor 1610 may control the transceiver 1630 to communicate with other devices, specifically, may send information or data to other devices, or receive information or data sent by other devices.

[0221] The transceiver 1630 may include a transmitter and a receiver. The transceiver 1630 may further include an antenna, and the number of antennas may be one or more.

[0222] Optionally, the processing device 1600 may specifically be a server of an embodiment of the present application, and the processing device 1600 may implement corresponding processes implemented by the server in various methods of the embodiment of the present application, which will not be described in detail here for the sake of brevity.

[0223] Optionally, the processing device 1600 may specifically be a terminal in an embodiment of the present application, and the processing device 1600 may implement corresponding processes implemented by the terminal in various methods in the embodiments of the present application, which will not be described in detail here for the sake of brevity.

[0224] Fig.17 It is a schematic structural diagram of the chip of an embodiment of the present application. Fig.17 The chip 1700 shown includes a processor 1710, which can call and run a computer program from a memory to implement the method in the embodiment of the present application.

[0225] Alternatively, if Fig.17 As shown, the chip 1700 may further include a memory 1720. The processor 1710 may call and run a computer program from the memory 1720 to implement the method in the embodiment of the present application.

[0226] The memory 1720 may be a separate device independent of the processor 1710 , or may be integrated into the processor 1710 .

[0227] Optionally, the chip 1700 may further include an input interface 1730. The processor 1710 may control the input interface 1730 to communicate with other devices or chips, and specifically, may obtain information or data sent by other devices or chips.

[0228] Optionally, the chip 1700 may further include an output interface 1740. The processor 1710 may control the output interface 1740 to communicate with other devices or chips, and specifically, may output information or data to other devices or chips.

[0229] Optionally, the chip can be applied to the server in the embodiments of the present application, and the chip can implement the corresponding processes implemented by the server in each method of the embodiments of the present application, which will not be described here for the sake of brevity.

[0230] Optionally, the chip can be applied to the terminal in the embodiments of the present application, and the chip can implement the corresponding processes implemented by the terminal in the various methods in the embodiments of the present application. For the sake of brevity, they will not be repeated here.

[0231] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0232] It should be understood that the processor of the embodiment of the present application may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method embodiment can be completed by the hardware integrated logic circuit or software instructions in the processor. The above processor can be a general processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to perform, or the hardware and software modules in the decoding processor are combined and performed. The software module can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, and other mature storage media in the art. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0233] It can be understood that the memory in the embodiments of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0234] It should be understood that the above-mentioned memory is exemplary but not restrictive. For example, the memory in the embodiments of the present application may also be static random access memory (static RAM, SRAM), dynamic random access memory (dynamic RAM, DRAM), synchronous dynamic random access memory (synchronous DRAM, SDRAM), double data rate synchronous dynamic random access memory (double data rate SDRAM, DDR SDRAM), enhanced synchronous dynamic random access memory (enhanced SDRAM, ESDRAM), synchronous link dynamic random access memory (synch link DRAM, SLDRAM) and direct memory bus random access memory (Direct Rambus RAM, DR RAM), etc. That is to say, the memory in the embodiments of the present application is intended to include but not limited to these and any other suitable types of memory.

[0235] An embodiment of the present application also provides a computer-readable storage medium for storing a computer program.

[0236] Optionally, the computer-readable storage medium can be applied to the server in the embodiments of the present application, and the computer program enables the computer to execute the corresponding processes implemented by the server in the various methods of the embodiments of the present application. For the sake of brevity, they are not repeated here.

[0237] Optionally, the computer-readable storage medium can be applied to the terminal in the embodiments of the present application, and the computer program enables the computer to execute the corresponding processes implemented by the terminal in the various methods of the embodiments of the present application. For the sake of brevity, they are not repeated here.

[0238] An embodiment of the present application also provides a computer program product, including computer program instructions.

[0239] Optionally, the computer program product can be applied to the server in the embodiments of the present application, and the computer program instructions enable the computer to execute the corresponding processes implemented by the server in the various methods of the embodiments of the present application. For the sake of brevity, they are not repeated here.

[0240] Optionally, the computer program product can be applied to the terminal in the embodiments of the present application, and the computer program instructions enable the computer to execute the corresponding processes implemented by the terminal in the various methods of the embodiments of the present application. For the sake of brevity, they are not repeated here.

[0241] The embodiment of the present application also provides a computer program.

[0242] Optionally, the computer program can be applied to the server in the embodiments of the present application. When the computer program runs on a computer, the computer executes the corresponding processes implemented by the server in the various methods of the embodiments of the present application. For the sake of brevity, they are not described here.

[0243] Optionally, the computer program can be applied to the terminal in the embodiments of the present application. When the computer program runs on the computer, the computer executes the corresponding processes implemented by the terminal in the various methods of the embodiments of the present application. For the sake of brevity, they are not repeated here.

[0244] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0245] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0246] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0247] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0248] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0249] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a server, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0250] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An image processing method, characterized in that: The method comprises: Calling the first model to analyze the target image and obtain an analysis result; the target image is an image captured by the terminal; the analysis result includes N subtasks required for the execution of the target task, where N is a positive integer; For each of the N subtasks, calling a second model matching the subtask to process the subtask and obtain a processing result; The processing results of the N subtasks and / or the operation information determined based on the processing results of the N subtasks are sent to the terminal for presentation.

2. The method according to claim 1, characterized in that The calling of the first model to analyze the target image includes: Call the first model to analyze the target image as follows: Identify key information of the target image, and determine scene information corresponding to the target image according to the key information; Determining N audit points associated with the target image based on scene information corresponding to the target image, wherein the N audit points have a corresponding relationship with the N subtasks; Performing intention recognition based on scene information corresponding to the target image and N audit points associated with the target image to obtain an intention recognition result; the intention recognition result represents the target task; Determine N subtasks for completing the target task based on the intention recognition result.

3. The method according to claim 2, characterized in that The key information for identifying the target image includes: Extracting a feature vector of the target image and converting the feature vector of the target image into text; wherein the feature vector represents a semantic feature of the target image; The key information in the text is extracted.

4. The method according to claim 2, characterized in that: The performing intention recognition based on the scene information corresponding to the target image and the N audit points associated with the target image includes: Obtain user instructions collected by the terminal; The user intention is determined based on the user instruction, and the intention is identified based on the user intention, scene information corresponding to the target image, and N audit points associated with the target image.

5. The method according to any one of claims 2 to 4, characterized in that The first model is a global model, and the second model is a local model; the global model refers to a model that has the ability to call the local model; different local models have different artificial intelligence AI capabilities; wherein the second model matching the subtask refers to a local model that has the AI ​​capability to process the subtask.

6. The method according to any one of claims 2 to 4, characterized in that The target task includes at least one of the following: a machine room inspection task, a construction site survey task, an equipment acceptance task, and an equipment maintenance task.

7. The method according to any one of claims 2 to 4, characterized in that The target image is an image of a real scene collected by the terminal.

8. The method according to any one of claims 1 to 4, characterized in that The method further comprises: Obtain the video of the real scene captured by the terminal; Detecting a key area and a key time period in the video; the key area refers to an area in the video where a target object and / or a target action exists; the key time period refers to a time period in the video where a target object and / or a target action exists; A key frame is extracted from the video based on the key area and the key time period, and the key frame is used as the target image.

9. The method according to any one of claims 1 to 4, characterized in that The calling of a second model matching the subtask to process the subtask includes: If there are multiple second models matching the subtask, a second model is selected from the multiple matching second models based on the model deployment location to process the subtask.

10. The method according to any one of claims 1 to 4, characterized in that The method further comprises: Training data is generated according to the target image and the processing results of the N subtasks, and the first model is trained using the training data.

11. The method according to any one of claims 1 to 4, characterized in that The method further comprises: The processing results of the N subtasks and / or the operation information determined based on the processing results of the N subtasks are sent to a remote terminal corresponding to the terminal for presentation.

12. An image processing method, characterized in that: The method comprises: Collect an image and send the image to a server; the image is used by the server to obtain the processing results of N subtasks corresponding to the target task by calling the model, where N is a positive integer; Receiving the processing results of the N subtasks and / or operation information determined based on the processing results of the N subtasks sent by the server; A first virtual screen is displayed superimposed on the real scene, and the processing result and / or the operation information is presented on the first virtual screen.

13. The method according to claim 12, characterized in that The processing result is associated with a target object in the real scene; The superimposing and displaying the first virtual picture in the real scene includes: A label is displayed at a first position relative to the target object in a real scene through a virtual display effect, wherein the label is used to indicate the target object; and the processing result and / or the operation information is displayed at a second position relative to the label through a virtual display effect.

14. The method according to claim 12, characterized in that The method further comprises: Sending the image to a remote terminal; the image is used by a remote user to provide operation suggestions on the remote terminal; receiving an operation suggestion sent by the remote terminal; A second virtual screen is displayed superimposed on the real scene, and the operation suggestion is presented on the second virtual screen.

15. The method according to any one of claims 12 to 14, characterized in that The method comprises: Based on the image, identifying information of a target object in the real scene; A third virtual picture is superimposed and displayed in the real scene, and information of the target object is presented on the third virtual picture.

16. The method according to any one of claims 12 to 14, characterized in that The method comprises: Receiving a task list sent by the server, wherein the task list includes one or more target tasks; A fourth virtual screen is displayed in a superimposed manner in the real scene, and one or more of the following contents are presented on the fourth virtual screen: the task list, the executed target tasks in the task list, and the unexecuted target tasks in the task list.

17. The method according to any one of claims 12 to 14, characterized in that The image is an image of the real scene.

18. An image processing device, characterized in that: The device comprises: A first calling unit is used to call the first model to analyze a target image and obtain an analysis result; the target image is an image captured by the terminal; the analysis result includes N subtasks required for the execution of the target task, where N is a positive integer; A second calling unit is used for calling a second model matching the subtask to process each of the N subtasks to obtain a processing result; The first communication unit is used to send the processing results of the N subtasks and / or the operation information determined based on the processing results of the N subtasks to the terminal for presentation.

19. An image processing device, characterized in that: The device comprises: An acquisition unit, used for acquiring images; A second communication unit is used to send the image to a server; the image is used by the server to obtain the processing results of N subtasks corresponding to the target task by calling the model, where N is a positive integer; and receive the processing results of the N subtasks sent by the server and / or operation information determined based on the processing results of the N subtasks; The display unit is used to display a first virtual screen superimposed on a real scene, wherein the processing result and / or the operation information is presented on the first virtual screen.

20. A processing device, characterized in that include: A processor and a memory, the memory being used to store a computer program, the processor being used to call and run the computer program stored in the memory to execute the method according to any one of claims 1 to 17.

21. A computer-readable storage medium, characterized in that: Used to store a computer program, the computer program causing a computer to execute the method according to any one of claims 1 to 17.

22. A computer program product, characterized in that The method comprises computer program instructions which cause a computer to execute the method as claimed in any one of claims 1 to 17.

Images