Visual recognition algorithm generation method and device based on natural language interaction and medium

Abstract

The invention provides a visual recognition algorithm generation method and device based on natural language interaction and a medium. The method comprises the steps of obtaining an input natural language interaction instruction; determining a corresponding target camera device, a visual identification scene, a visual identification task and a visual identification early warning condition based on the natural language interaction instruction by using a preset large language model; splitting the visual identification task into a plurality of sub-tasks with a logical dependency relationship; determining a target basic algorithm matched with each subtask; taking the target basic algorithms matched with the plurality of sub-tasks as the target basic algorithms matched with the visual identification tasks; and generating a target visual identification algorithm based on the plurality of target basic algorithms matched with the visual identification task, the visual identification scene and the visual identification early warning condition, and applying the target visual identification algorithm to the target camera device. According to the invention, the generation efficiency of the visual identification algorithm can be improved, the generation cost of the visual identification algorithm is lower, and the use effect is better.

Description

Technical Field

[0001] This application relates to the field of visual recognition algorithm technology, specifically to a method, device, and medium for generating visual recognition algorithms based on natural language interaction. Background Technology

[0002] With the deepening application of artificial intelligence technology in the field of video surveillance, visual recognition algorithms have been widely deployed in smart parks, intelligent manufacturing, public security and other scenarios.

[0003] In related technologies, visual recognition algorithms are usually generated by manual programming. However, manual programming is inefficient and cannot adapt to rapidly changing needs, resulting in excessively high generation costs and poor performance of visual recognition algorithms. Summary of the Invention

[0004] The embodiments of this application provide a method, device, and medium for generating visual recognition algorithms based on natural language interaction, aiming to improve the generation efficiency of visual recognition algorithms.

[0005] In a first aspect, embodiments of this application provide a method for generating a visual recognition algorithm based on natural language interaction, the method comprising:

[0006] Obtain input natural language interaction commands;

[0007] Using a pre-defined large language model, the corresponding target camera device, visual recognition scene, visual recognition task, and visual recognition warning conditions are determined based on the natural language interaction instructions.

[0008] The visual recognition task is broken down into multiple sub-tasks with logical dependencies.

[0009] Determine the target base algorithm for matching each of the subtasks;

[0010] The target base algorithms for matching multiple sub-tasks are all used as the target base algorithms for matching the visual recognition task.

[0011] Based on the multiple target-based algorithms for matching the visual recognition task, the visual recognition scene, and the visual recognition warning conditions, a target visual recognition algorithm is generated, and the target visual recognition algorithm is applied to the target camera device.

[0012] In the above embodiments, a preset large language model is used to determine the target camera device, visual recognition scene, visual recognition task, and visual recognition warning conditions based on natural language interaction instructions. The visual recognition task is then broken down into multiple sub-tasks with logical dependencies to match the target base algorithm. Based on multiple target base algorithms, the visual recognition scene, and the visual recognition warning conditions, a target visual recognition algorithm applied to the target camera device is directly generated, improving the generation efficiency of the visual recognition algorithm, making the generation cost of the visual recognition algorithm lower, and the usage effect better.

[0013] In one embodiment, the target underlying algorithm for determining the matching of each of the subtasks includes:

[0014] For each subtask, check in the preset basic algorithm database whether there is a preset basic algorithm that matches the subtask;

[0015] If a preset basic algorithm for subtask matching exists in the basic algorithm database, then the target basic algorithm for subtask matching is determined based on the preset basic algorithm for subtask matching in the basic algorithm database.

[0016] If the preset basic algorithm for matching the subtask does not exist in the basic algorithm database, a target basic algorithm for matching the subtask is generated based on the zero-sample mechanism.

[0017] In the above embodiments, for each subtask, a matching preset basic algorithm is detected in the preset basic algorithm database. If the matching preset basic algorithm exists, the target basic algorithm is determined based on the database content. If the matching preset basic algorithm does not exist, the target basic algorithm is generated based on zero-sample capability, ensuring that each subtask can obtain a matching target basic algorithm.

[0018] In one embodiment, determining the target basic algorithm for subtask matching based on the preset basic algorithm for subtask matching in the basic algorithm database includes:

[0019] If the number of preset basic algorithms for subtask matching in the basic algorithm database is one, then the preset basic algorithm for subtask matching in the basic algorithm database shall be used as the target basic algorithm for subtask matching.

[0020] If there are multiple preset basic algorithms for subtask matching in the basic algorithm database, then based on the computing resources of the target camera device, a corresponding target value for computing resource consumption is determined. Based on the target value for computing resource consumption, the target basic algorithm for subtask matching is determined from the multiple preset basic algorithms for subtask matching in the basic algorithm database.

[0021] In the above embodiments, when there are multiple matching preset basic algorithms in the preset basic algorithm database, the target value of computing resource consumption is determined based on the computing resource quantity of the target camera device, and the target basic algorithm is determined from the multiple preset basic algorithms accordingly, so that the selected algorithm is adapted to the hardware resource capabilities of the target camera device.

[0022] In one embodiment, the target basic algorithm for generating the subtask matching based on the zero-shot mechanism includes:

[0023] Obtain the target object label corresponding to the subtask;

[0024] Based on the target object label, generate visual cue description information;

[0025] Using a preset text encoder, feature mapping is performed on the visual cue description information to obtain a text feature vector;

[0026] The text feature vectors are configured into the feature matching layer of a pre-trained open vocabulary visual recognition model;

[0027] The open vocabulary visual recognition model configured with the text feature vectors will be used as the target base algorithm for the subtask matching.

[0028] In the above embodiments, by obtaining the target object label of the subtask and generating visual cue description information, the text feature vector is obtained using a preset text encoder and configured into the feature matching layer of the pre-trained open vocabulary visual recognition model, thereby directly constructing the target basic algorithm that matches the subtask.

[0029] In one embodiment, the generation of a target visual recognition algorithm based on multiple target-based algorithms for matching the visual recognition task, the visual recognition scene, and the visual recognition warning conditions includes:

[0030] Based on the logical dependency relationship, multiple target basic algorithms for matching the visual recognition task are combined to obtain a target combination algorithm;

[0031] The target visual recognition algorithm is generated based on the target combination algorithm, the visual recognition scene, and the visual recognition warning conditions.

[0032] In the above embodiments, multiple target basic algorithms are combined based on logical dependencies to obtain a target combination algorithm, and a target visual recognition algorithm is generated by combining visual recognition scenarios and visual recognition early warning conditions, thus integrating discrete algorithm units into a complete algorithm with scene adaptability and early warning capabilities.

[0033] In one embodiment, the combination of multiple target-based algorithms for matching the visual recognition task based on the logical dependency relationship to obtain a target combination algorithm includes:

[0034] Based on the aforementioned logical dependencies, the execution topology sequence of multiple target base algorithms for matching the visual recognition task is determined;

[0035] Determine the output parameters of the target base algorithm located upstream in the execution topology sequence, and the input parameters of the target base algorithm located downstream;

[0036] Establish a data mapping channel between the output parameters and the input parameters;

[0037] The target combination algorithm is generated based on the execution topology sequence and the data mapping channel.

[0038] In the above embodiments, the execution topology sequence is determined based on logical dependencies, and a data mapping channel is established between the output parameters of the upstream target basic algorithm and the input parameters of the downstream target basic algorithm to ensure accurate data flow and execution order among the algorithm nodes in the target combination algorithm.

[0039] In one embodiment, generating the target visual recognition algorithm based on the target combination algorithm, the visual recognition scene, and the visual recognition warning conditions includes:

[0040] In the preset scene configuration mapping table, the image enhancement strategy and result filtering strategy for matching the visual recognition scene are determined;

[0041] The image enhancement strategy is determined as a visual preprocessing algorithm, and the result filtering strategy is determined as a visual postprocessing algorithm.

[0042] A generation algorithm execution pipeline is provided, wherein, in the algorithm execution pipeline, the data output port of the visual preprocessing algorithm is connected to the data input port of the target combination algorithm, and the data output port of the target combination algorithm is connected to the data input port of the visual postprocessing algorithm;

[0043] The visual recognition warning conditions are configured into the logical decision layer of the algorithm execution pipeline to obtain the target visual recognition algorithm.

[0044] In the above embodiments, visual preprocessing algorithms and visual postprocessing algorithms are determined based on the visual recognition scenario, and they are connected with the target combination algorithm in the algorithm execution pipeline. Visual recognition early warning conditions are configured to the logic decision layer, and the generated algorithm execution pipeline has the ability to perform full-process image processing and business logic judgment.

[0045] In one embodiment, after the generation algorithm execution pipeline, the following is further included:

[0046] Based on the resource consumption attributes of each algorithm node in the algorithm execution pipeline, the total resource consumption value of the algorithm is determined;

[0047] Obtain the current available resource threshold of the target camera device;

[0048] If the total resource consumption of the algorithm is greater than the current available resource threshold, then the algorithm execution pipeline is pruned based on the visual recognition scenario.

[0049] In the above embodiments, when the total resource consumption of the algorithm is greater than the current available resource threshold of the target camera device, the algorithm execution pipeline is pruned based on the visual recognition scene so that the generated algorithm execution pipeline can be adapted to the current actual available resources of the target camera device.

[0050] Secondly, embodiments of this application provide an electronic device including a processor and a memory, wherein the memory stores a computer program configured to be executed by the processor to implement the visual recognition algorithm generation method based on natural language interaction as described in any of the preceding claims.

[0051] Thirdly, embodiments of this application provide a computer-readable storage medium storing a computer program configured to be executed by a processor to implement the visual recognition algorithm generation method based on natural language interaction as described in any of the preceding claims.

[0052] The beneficial effects of the embodiments of this application are as follows:

[0053] In the embodiments of this application, a preset large language model is used to determine the corresponding target camera device, visual recognition scene, visual recognition task, and visual recognition warning conditions based on natural language interaction instructions. Then, the target basic algorithm is determined, and a target visual recognition algorithm applied to the target camera device is generated. This realizes the automation of the visual recognition algorithm generation process, thereby improving the generation efficiency of the visual recognition algorithm, making the generation cost of the visual recognition algorithm lower, and the usage effect better. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0055] Figure 1This is a schematic flowchart of an embodiment of the visual recognition algorithm generation method based on natural language interaction provided in this application. Detailed Implementation

[0056] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. In addition, in the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0057] Firstly, embodiments of this application provide a method for generating visual recognition algorithms based on natural language interaction, wherein the executing entity is a system for generating visual recognition algorithms based on natural language interaction (hereinafter referred to as the "system"). Specifically, refer to... Figure 1 A visual recognition algorithm generation method based on natural language interaction may include:

[0058] S101. Obtain the input natural language interaction instructions.

[0059] In the embodiments of this application, natural language interaction instructions refer to unstructured or semi-structured information input by the user through a human-computer interaction interface to express visual recognition business needs. These instructions carry the user's subjective intent regarding a specific monitoring scenario, monitoring object, and alarm rules. The acquisition process typically relies on the data transmission link between the front-end interactive terminal and the back-end server, supporting real-time data reception via communication protocols such as Hypertext Transfer Protocol (HTTP) or WebSocket.

[0060] In some embodiments of this application, the input modalities of natural language interaction instructions include text modalities, speech modalities, and visual-language fusion modalities. For the text modality, the system receives a character sequence; for the speech modality, the system calls an Automatic Speech Recognition (ASR) component to convert acoustic signals into text sequences; for the visual-language fusion modality, the system receives a combination of text instructions and spatial selection operations (such as box selection on an electronic map or video frame), and embeds spatial coordinate parameters into the metadata of the text instructions through coordinate mapping technology to clarify the spatial referent in the instructions.

[0061] S102. Using a pre-set large language model, based on natural language interaction instructions, determine the corresponding target camera device, visual recognition scene, visual recognition task, and visual recognition warning conditions.

[0062] In the embodiments of this application, the pre-defined Large Language Model (LLM) refers to a deep neural network model that has been pre-trained on massive amounts of text data and fine-tuned for vertical fields such as security monitoring and industrial quality inspection, possessing semantic understanding, intent recognition, and entity extraction capabilities. The process of determining the above parameters is essentially the process of using the Large Language Model to map unstructured natural language into structured configuration parameters.

[0063] In some embodiments of this application, this step specifically includes using prompt engineering techniques to construct prompt words containing task descriptions, example samples, and output format specifications, and inputting natural language interaction instructions into a large language model. The large language model performs semantic parsing and slot filling operations:

[0064] Target camera device: refers to the hardware carrier that executes the visual recognition algorithm. The model extracts location terms or device names (such as "Warehouse No. 3" or "South Gate Camera") from the instructions and retrieves a unique matching device identifier (Device ID) from the device topology database.

[0065] Visual scene recognition: This refers to the type of environment in which the camera device is located, used to determine the domain adaptation strategy for image processing. The model classifies scene labels based on environmental descriptors (such as "night," "rainy day," "indoors").

[0066] Visual recognition task: refers to the core business objective that the user expects to achieve. The model recognizes the verb-object combination in the instruction (such as "detect smoking" or "count traffic flow") to determine the main functional category of the algorithm.

[0067] Visual recognition warning conditions refer to the logical judgment thresholds that trigger a business response. The model extracts quantifiers, time words, or logical relation words from the instructions and quantifies them into specific numerical parameters (such as "stay time > 30 seconds", "number of people > 5").

[0068] S103. The visual recognition task is broken down into multiple sub-tasks with logical dependencies.

[0069] In the embodiments of this application, a subtask refers to an atomic processing unit that constitutes a complex visual recognition task, with a granularity corresponding to a single-function algorithm module. Logical dependency refers to the data input / output dependence or temporal sequence constraints that exist between multiple subtasks during execution. The decomposition process utilizes the logical reasoning capabilities of a large language model or the concept chain technique to break down the macroscopic business task into an executable sequence of steps.

[0070] In some embodiments of this application, logical dependencies are constructed as a directed acyclic graph (DAG) structure to accurately represent the execution flow:

[0071] Serial dependency: The input of the current subtask strongly depends on the output of the preceding subtask. For example, in the task of "identifying workers not wearing safety helmets", the "personnel target detection" subtask is the preceding node, the "head key point localization" subtask is the in-order node, and the "safety helmet wearing classification" subtask is the following node, and the data stream is passed sequentially.

[0072] Parallel execution: Multiple subtasks have no data dependencies and can be computed concurrently to reduce latency. For example, subtasks that simultaneously detect "flame" and "smoke" can be processed in parallel, and the results can be aggregated at the end.

[0073] Conditional branching: The subsequent path is determined based on the output of the preceding subtask. For example, the "license plate recognition" subtask is activated only if the "vehicle detected" subtask outputs true.

[0074] S104. Determine the target basic algorithm for matching each subtask.

[0075] In the embodiments of this application, the target basic algorithm refers to a pre-packaged executable code unit or deep learning model file (such as YOLO series detection models, ResNet series classification models, etc.) with an independent interface. The matching process refers to finding the computational component in a pre-built algorithm repository that best matches the semantics of the subtask.

[0076] In some embodiments of this application, the system maintains an algorithm registry, recording the functional description, input / output specifications, and performance metrics of each basic algorithm. Utilizing the semantic embedding capabilities of a large language model, the cosine similarity between the text description vector of a subtask and the functional description vectors of each algorithm in the registry is calculated, and the basic algorithm with the highest similarity is selected as the target basic algorithm. Alternatively, a predefined rule mapping table can be used to directly index the corresponding specific algorithm version based on the category label of the subtask.

[0077] S105. The target base algorithms for matching multiple sub-tasks are all used as the target base algorithms for matching visual recognition tasks.

[0078] In the embodiments of this application, the system can perform redundancy checks and version consistency checks on the aggregated set of target basic algorithms. If multiple subtasks reuse the same basic algorithm (for example, if one subtask needs to extract features and another subtask also needs to extract the same features), the system will merge the computing nodes to avoid wasting computing power due to repeated inference and ensure that the generated set of target basic algorithms is the simplest computational graph.

[0079] S106. Based on multiple target-based algorithms for visual recognition task matching, visual recognition scenarios, and visual recognition early warning conditions, a target visual recognition algorithm is generated and applied to the target camera device.

[0080] In the embodiments of this application, generating a target visual recognition algorithm refers to orchestrating selected algorithm components, configuration parameters, and scene strategies into a complete, deployable application or inference pipeline. This target visual recognition algorithm can be directly deployed to edge computing nodes or cloud inference servers for execution.

[0081] In some embodiments of this application, the generation process of the target visual recognition algorithm includes the following steps:

[0082] Scene-adaptive configuration: Based on the determined visual recognition scene, the corresponding image preprocessing operator is automatically loaded. For example, if the scene is "low light", an "image enhancement algorithm" is inserted at the beginning of the pipeline; if the scene is "long distance", the upsampling parameters of the input resolution are configured.

[0083] Logical pipeline construction: Based on the logical dependencies determined in step S103, connect multiple target basic algorithms according to topological order. Define the data transfer format between different algorithm modules (such as tensor dimension transformation and normalization processing) to ensure interface compatibility.

[0084] Parameter injection and encapsulation: Visual recognition warning conditions are transformed into configuration parameters for logical judgment nodes (such as confidence thresholds and IOU thresholds). Finally, the entire processing flow is encapsulated into a containerized image (Docker Image) or a specific format inference configuration file and deployed to computing resources bound to the target camera device to achieve an end-to-end business closed loop.

[0085] As can be seen, the embodiments of this application utilize a preset large language model to determine the target camera device, visual recognition scene, visual recognition task, and visual recognition warning conditions based on natural language interaction instructions. The visual recognition task is then broken down into multiple sub-tasks with logical dependencies to match the target basic algorithm. Based on multiple target basic algorithms, the visual recognition scene, and the visual recognition warning conditions, a target visual recognition algorithm applied to the target camera device is directly generated, improving the generation efficiency of the visual recognition algorithm, making the generation cost of the visual recognition algorithm lower, and the usage effect better.

[0086] In some embodiments of this application, the target underlying algorithm for determining the matching of each subtask includes:

[0087] S201. For each subtask, check in the preset basic algorithm database whether there is a preset basic algorithm that matches the subtask.

[0088] In the embodiments of this application, the preset basic algorithm database refers to a structured storage system that stores several standard algorithm models trained, verified, and optimized using data from relevant fields. Each entry in this database is a preset basic algorithm, which includes a model weight file, an inference configuration file, and metadata tags. The metadata tags include, but are not limited to, the algorithm's functional description (e.g., "helmet wearing detection"), applicable scenarios (e.g., "indoor lighting"), performance metrics (e.g., average accuracy, frames per second (FPS), and hardware resource requirements (e.g., video memory usage, computing power requirements).

[0089] In some embodiments of this application, the detection process employs a semantic vector-based retrieval technique. The system first calls a text embedding model to convert the natural language description of the subtask into a high-dimensional subtask semantic vector. Simultaneously, the functional descriptions of each preset basic algorithm in the preset basic algorithm database have been pre-converted into algorithm feature vectors and a vector index has been constructed. The system calculates the cosine similarity between the subtask semantic vector and each algorithm feature vector. If an algorithm feature vector has a cosine similarity greater than a preset matching threshold, it is determined that a preset basic algorithm matching that subtask exists. This approach effectively handles synonyms, near-synonyms, or sentence structure variations, avoiding the limitations of precise keyword matching in related technologies.

[0090] S202. If there is a preset basic algorithm for subtask matching in the basic algorithm database, then determine the target basic algorithm for subtask matching based on the preset basic algorithm for subtask matching in the basic algorithm database.

[0091] In the embodiments of this application, determining the target basic algorithm means selecting the algorithm instance that is most suitable for the current execution environment from the retrieved candidate preset basic algorithms.

[0092] In some embodiments of this application, when multiple pre-defined basic algorithms with matching functions are retrieved (e.g., "face detection - high-precision version" and "face detection - lightweight version"), the system adaptively selects the appropriate algorithm based on the hardware attributes of the target camera device determined in previous steps. If the target camera device is an edge camera with limited computing power, the system selects the lightweight version with lower computational load and faster inference speed as the target basic algorithm; if the target camera device is associated with a high-performance cloud computing cluster, the system selects the high-precision version with higher accuracy as the target basic algorithm. By matching not only functional semantics but also computing resource constraints, the system ensures that the determined target basic algorithm can be effectively deployed and executed.

[0093] S203. If there is no preset basic algorithm for subtask matching in the basic algorithm database, then a target basic algorithm for subtask matching is generated based on the zero-sample mechanism.

[0094] In the embodiments of this application, the zero-shot mechanism refers to a technique that utilizes the open vocabulary recognition capability of a large-scale pre-trained multimodal model to identify visual objects that have never appeared in the training set without the need for retraining or fine-tuning for a specific category.

[0095] In some embodiments of this application, the specific steps of generating the target base algorithm based on the zero-shot mechanism include:

[0096] Cue word extraction and construction: A large language model is used to perform dependency parsing on the text of the subtask to extract noun phrases representing the core objects to be identified (e.g., extracting "red backpack" from "detect if anyone left a red backpack"). This noun phrase is then filled into a predefined cue template (e.g., "a photo containing [red backpack]") to generate text cue words for visual matching.

[0097] Visual-Language Model Configuration: Invoke a pre-built open-vocabulary object detection model, such as a model based on a contrastive language-image pre-trained architecture. Input the generated text prompts into the model's text encoder to generate text feature embeddings.

[0098] Algorithm Encapsulation: The inference engine of the open-vocabulary object detection model and the generated text feature embeddings are bound and encapsulated into an independent executable unit. This executable unit is the underlying algorithm for the generated subtask matching. During runtime, this underlying algorithm identifies specific targets by calculating the matching degree between the input image region features and the text feature embeddings.

[0099] As can be seen, in this embodiment of the application, for each subtask, the system checks whether a matching preset basic algorithm exists in the preset basic algorithm database. If the algorithm exists, the target basic algorithm is determined based on the database content. If the algorithm does not exist, the target basic algorithm is generated based on zero-sample capability, ensuring that each subtask can obtain a matching target basic algorithm.

[0100] In some embodiments of this application, the target basic algorithm for subtask matching is determined based on a preset basic algorithm for subtask matching in the basic algorithm database, including:

[0101] S301. If the number of preset basic algorithms for subtask matching in the basic algorithm database is one, then the preset basic algorithm for subtask matching in the basic algorithm database shall be used as the target basic algorithm for subtask matching.

[0102] In the embodiments of this application, this situation typically occurs in sub-task scenarios where the functional definition is highly standardized or specialized. When the preset basic algorithm database contains only one verified stable version for a specific sub-task (e.g., "license plate area positioning" or "QR code format decoding"), the preset basic algorithm is the only applicable algorithm.

[0103] In some embodiments of this application, the system directly calls the configuration parameters of the unique preset basic algorithm and locks it as the target basic algorithm. This processing logic eliminates complex decision-making overhead, ensures low latency and high determinism for standard functional module calls, and maintains the simplicity of the algorithm generation process.

[0104] S302. If there are multiple preset basic algorithms for subtask matching in the basic algorithm database, then based on the computing resources of the target camera device, determine the corresponding target value for computing resource consumption, and based on the target value for computing resource consumption, determine the target basic algorithm for subtask matching from the multiple preset basic algorithms for subtask matching in the basic algorithm database.

[0105] In the embodiments of this application, multiple matching preset base algorithms refer to multiple algorithm variants stored in the database for the same subtask function (e.g., "pedestrian detection"), based on different neural network architectures (e.g., ResNet, MobileNet, ShuffleNet), different parameter levels (e.g., Nano, Small, Large versions), or different quantization precisions (e.g., FP32, FP16, INT8). These variants exhibit different trade-offs between detection accuracy and computational cost.

[0106] In the embodiments of this application, the computing resources of the target camera device refer to the upper limit of the physical capabilities of the hardware device for performing edge computing tasks and its current availability. This indicator includes, but is not limited to, the number of floating-point operations per second (FLOPS), video memory capacity, memory bandwidth, and thermal design power (TDP) of the graphics processor or neural processing unit (NPU).

[0107] In some embodiments of this application, the process of determining the corresponding target value for computing resource consumption includes: the system first obtains the hardware specifications and current load status of the target camera device through a network management protocol (such as Simple Network Management Protocol SNMP) or an edge agent. Then, based on the remaining idle computing resources and a preset resource reservation strategy, the system calculates the maximum resource quota allowed for the current subtask. This maximum resource quota is the target value for computing resource consumption, which can be represented as a multi-dimensional vector containing the maximum allowed number of floating-point operations per second and the maximum allowed video memory usage.

[0108] In some embodiments of this application, the process of determining the target basic algorithm based on the target value of computational resource consumption adopts a "constrained optimal solution search" strategy. The system traverses all matching preset basic algorithms and reads the resource requirement characteristics recorded in the metadata of each preset basic algorithm. First, the system filters out all candidate algorithms whose resource requirement characteristics exceed the target value of computational resource consumption, obtaining a feasible solution set. Then, in the feasible solution set, the system selects the preset basic algorithm with the best performance index as the target basic algorithm based on preset performance priority indicators (such as average accuracy). For example, if the target camera device is a low-power camera with limited computing power and a low target value of computational resource consumption, the system will automatically eliminate the high-computing "large server version model" from multiple variants of "personnel detection" and select the version with the highest accuracy from the remaining "lightweight mobile model". This method ensures that the generated algorithm will not cause system crashes or frame rate stuttering due to exceeding the hardware load, while making full use of hardware performance to provide the best recognition effect.

[0109] As can be seen, in the embodiments of this application, when there are multiple matching preset basic algorithms in the preset basic algorithm database, the target value of computing resource consumption is determined based on the computing resource quantity of the target camera device, and the target basic algorithm is determined from the multiple preset basic algorithms accordingly, so that the selected algorithm is adapted to the hardware resource capabilities of the target camera device.

[0110] In some embodiments of this application, a target-based algorithm for generating subtask matching is generated based on a zero-shot mechanism, including:

[0111] S401. Obtain the target object label corresponding to the subtask.

[0112] In the embodiments of this application, the target object label refers to a semantic keyword extracted from the natural language text description of the subtask that can uniquely represent the visual entity to be identified. This label is usually a specific noun or noun phrase, serving as a core anchor connecting the language modality and the visual modality.

[0113] In some embodiments of this application, the acquisition process utilizes part-of-speech tagging and dependency parsing algorithms from natural language processing techniques. The system first segments the subtask text, identifying noun components. Then, based on the syntactic structure tree, it filters out direct objects that depend on verbs (such as "detect" and "identify") as candidate entities. For example, for the subtask "confirm whether any staff members are not wearing reflective vests," the system identifies "reflective vest" as the core visual object of interest and determines it as the target object label. This process ensures the accuracy of label extraction and filters out interference from descriptive words.

[0114] S402. Generate visual cues and descriptions based on the target object's label.

[0115] In the embodiments of this application, visual cue description information refers to a set of natural language text sequences constructed using Prompt Engineering techniques to guide a multimodal model in understanding the context of a target object.

[0116] In some embodiments of this application, the generation process employs a template-filling strategy. The system has a pre-set set of prompt templates containing different scenario assumptions (e.g., "a photo of [label]", "the outline of [label]", "[label] from a surveillance perspective"). The system fills the target object label obtained in S401 into the above-mentioned multiple templates, generating multiple parallel visual prompt descriptions. Compared to a single description, using diverse templates can cover the semantic features of the target object under different shooting angles, occlusion conditions, or lighting conditions, thereby enhancing the generalization representation capability of the subsequently generated text feature vectors.

[0117] S403. Using a preset text encoder, perform feature mapping on the visual cue description information to obtain a text feature vector.

[0118] In the embodiments of this application, the preset text encoder is the text processing branch in a pre-trained open-vocabulary visual recognition model (such as Contrastive Language-Image Pre-training, CLIP), typically employing a Transformer architecture. A text feature vector refers to a numerical vector located in a high-dimensional feature space, which is semantically aligned with the visual feature vector of the target object.

[0119] In some embodiments of this application, the system inputs each generated visual cue description into a text encoder to calculate the corresponding primary feature vector. Subsequently, the system performs normalization and average pooling operations on these primary feature vectors to synthesize a final text feature vector. This operation effectively smooths out semantic biases that may arise from a single description, resulting in a more robust text feature vector that represents the category center of the target object.

[0120] S404. Configure the text feature vector into the feature matching layer of the pre-trained open vocabulary visual recognition model.

[0121] In the embodiments of this application, the feature matching layer is a functional module in the open vocabulary visual recognition model used to calculate the similarity (such as cosine similarity or dot product) between image region features and text semantic features. The configuration process is essentially dynamically constructing the weight matrix of the classifier.

[0122] In some embodiments of this application, this step specifically involves using the text feature vector obtained in S403 as classification weights, replacing or appending it to the model's classification head. For object detection tasks, the system loads the text feature vector into the subsequent classification branch of the Region Proposal Network. During the inference phase, the model extracts candidate region features from the image and performs a matching operation with the text feature vector. If the similarity exceeds a preset threshold, the region is determined to contain the target object. This dynamic configuration method allows the model to recognize specific new categories without retraining.

[0123] S405. An open vocabulary visual recognition model configured with text feature vectors is used as the target basic algorithm for subtask matching.

[0124] In the embodiments of this application, this step is a process of standardizing and encapsulating dynamically constructed model instances.

[0125] In some embodiments of this application, the system packages an open-vocabulary visual recognition model loaded with specific text feature vectors, along with its image preprocessing pipeline (such as scaling and normalization parameters) and post-processing logic (such as Non-Maximum Suppression, NMS), into a standalone executable container or function interface. This encapsulated whole constitutes the target basic algorithm for the current subtask. This target basic algorithm exposes standard input / output interfaces and can be integrated into the execution flow of the entire visual recognition algorithm, thus completing the transformation from abstract language description to concrete executable algorithm code.

[0126] As can be seen, the embodiments of this application obtain the target object label of the subtask and generate visual cue description information, use a preset text encoder to obtain the text feature vector and configure it to the feature matching layer of the pre-trained open vocabulary visual recognition model, and directly construct the target basic algorithm that matches the subtask.

[0127] In some embodiments of this application, a target visual recognition algorithm is generated based on multiple target-based algorithms for visual recognition task matching, visual recognition scenarios, and visual recognition early warning conditions, including:

[0128] S501. Based on logical dependencies, multiple target-based algorithms for visual recognition task matching are combined to obtain a target combination algorithm.

[0129] In the embodiments of this application, logical dependencies are directed acyclic graph structures that define the data transfer and execution order among multiple subtasks. The objective combination algorithm is a formally described operator pipeline. This objective combination algorithm encapsulates discrete objective base algorithms into computational nodes interconnected through input / output interfaces.

[0130] In some embodiments of this application, the combination process is implemented using a graph-based visual programming paradigm. The system parses the logical dependencies established in the preceding steps and instantiates the corresponding target basic algorithm for each logical node. If there are parallel branches (e.g., "flame detection" and "smoke detection" in parallel), the system inserts a multiplexer during combination to distribute the video stream data; if there are conditional branches, the system inserts a logical judgment unit.

[0131] In some embodiments of this application, the system automatically generates a metadata file (e.g., in JSON format) describing the graph structure. This file explicitly specifies the algorithm model path, input tensor dimension, output tensor format, and index of the next node for each node. This metadata file constitutes the core description of the target combinatorial algorithm, ensuring the portability of the algorithm flow.

[0132] S502. Based on the target combination algorithm, visual recognition scene, and visual recognition early warning conditions, generate a target visual recognition algorithm.

[0133] In the embodiments of this application, generating a target visual recognition algorithm is the process of transforming an abstract algorithm flowchart into a specific deployable and executable binary program or container image.

[0134] In some embodiments of this application, the target combination algorithm is parameterized and optimized based on the visual recognition scene. The environmental features corresponding to the visual recognition scene (such as "low light at night" or "rainy / foggy weather") will trigger specific image preprocessing parameter configurations. For example, if the scene is "nighttime", the system automatically injects "histogram equalization" or "gamma correction" parameters into the preprocessing stage of the target combination algorithm and correspondingly lowers the confidence threshold of the target detection algorithm to adapt to low signal-to-noise ratio environments while ensuring recall.

[0135] In some embodiments of this application, a business logic layer is constructed based on visual recognition warning conditions. Visual recognition warning conditions (such as "staying time exceeds 30 seconds" or "more than 5 people in the area") are converted into specific rule-based judgment code. The system generates a post-processing script that subscribes to the output data stream of the target combination algorithm. For example, for the warning condition "more than 5 people," the script counts the number of detection boxes in real time; once the counter value exceeds 5, an alarm event is triggered.

[0136] In some embodiments of this application, the final generated target visual recognition algorithm is presented as a self-contained software distribution package. This distribution package integrates the model file of the basic target algorithm, the configuration file describing the execution flow, the preprocessing parameters for scene adaptation, and the judgment script for implementing the early warning logic. This distribution package can be directly distributed to the runtime environment of edge computing devices or cloud inference services, and upon startup, it can execute end-to-end visual monitoring tasks.

[0137] As can be seen, the embodiments of this application combine multiple target basic algorithms based on logical dependencies to obtain a target combination algorithm, and combine visual recognition scenarios and visual recognition early warning conditions to generate a target visual recognition algorithm, integrating discrete algorithm units into a complete algorithm with scene adaptability and early warning capabilities.

[0138] In some embodiments of this application, based on logical dependencies, multiple target-based algorithms for visual recognition task matching are combined to obtain a target combination algorithm, including:

[0139] S601. Based on logical dependencies, determine the execution topology sequence of multiple target-based algorithms for visual recognition task matching.

[0140] In the embodiments of this application, the logical dependency relationship is a directed acyclic graph that defines the predecessor and successor constraints between subtasks. The execution topology sequence refers to a linearized algorithm scheduling queue generated based on this directed acyclic graph, ensuring that the execution conditions (i.e., predecessor dependencies) of any node in the queue have been satisfied.

[0141] In some embodiments of this application, the determination process employs a topological sorting algorithm. The system traverses the graph structure of logical dependencies, calculating the in-degree of each node. First, nodes with an in-degree of zero (i.e., initial algorithms that do not depend on the output of other algorithms, such as "full-graph object detection") are added to the beginning of the sequence. Then, the node and its emanating directed edges are removed, the in-degree of the remaining nodes is updated, and this process is repeated until all nodes are added to the sequence. For example, for the task "detect vehicles first, then recognize license plates," topological sorting ensures that the generated execution topological sequence strictly follows the order of [vehicle detection algorithm, license plate recognition algorithm], thereby guaranteeing the causality of the data flow.

[0142] S602. Determine the output parameters of the target base algorithm located upstream in the execution topology sequence, and the input parameters of the target base algorithm located downstream.

[0143] In the embodiments of this application, the upstream target basic algorithm refers to the algorithm node that is earlier in the execution topology sequence, and its output data is used by subsequent nodes; the downstream target basic algorithm refers to the algorithm node that depends on the data of the preceding node. Output parameters and input parameters refer to the data fields declared by each algorithm module in the standardized interface definition language, including tensor dimensions, data types, and semantic labels.

[0144] In some embodiments of this application, each target underlying algorithm is associated with an interface description file. This file defines the algorithm's input and output parameters in a structured format, including parameter names, data types, data structures, and explanations of their meanings.

[0145] For example, the interface description file for a "person detection" algorithm might define its output parameter as `detection_results`, a list where each element is a dictionary containing bounding box coordinates and confidence values. The interface description file for a "face counting" algorithm might define its input parameter as `face_list`, also a list.

[0146] S603. Establish a data mapping channel between output parameters and input parameters.

[0147] In the embodiments of this application, the data mapping channel refers to the logical link that connects upstream and downstream algorithm nodes to achieve data transmission and format adaptation.

[0148] In some embodiments of this application, the establishment process includes field matching and data transformation. If the type definitions of the upstream output parameters and the downstream input parameters are completely consistent, the system directly establishes a memory-shared or reference-passing channel. If there is an inconsistency (e.g., the "detection algorithm" outputs full-image coordinates, while the "recognition algorithm" requires cropped image patches), the system automatically inserts a data adapter or middleware. This data adapter performs necessary preprocessing logic, such as cropping and scaling the Region of Interest (ROI) on the original video frame based on the bounding box coordinates output by the upstream, and using the processed image matrix as input to the downstream algorithm.

[0149] S604. Based on the execution topology sequence and data mapping channel, generate the target combination algorithm.

[0150] In the embodiments of this application, the target combination algorithm refers to an executable program or workflow definition file that encapsulates the calling logic of all algorithm nodes and the data flow rules.

[0151] In some embodiments of this application, the generation process employs code generation technology. Based on a defined execution topology sequence and data mapping channels, the system automatically synthesizes an orchestration script. This script sequentially instantiates each target basic algorithm object and assigns the output of the upstream object to the input parameters of the downstream object according to the mapping relationship. Alternatively, the system generates a configuration file conforming to a general workflow language standard for the workflow engine to parse and execute. This target combination algorithm not only includes the reasoning process of the algorithm model but also all necessary intermediate data processing logic, constituting a complete end-to-end processing pipeline.

[0152] As can be seen, the embodiments of this application determine the execution topology sequence based on logical dependency relationships, and establish a data mapping channel between the output parameters of the upstream target basic algorithm and the input parameters of the downstream target basic algorithm to ensure the accuracy of data flow and execution order between the algorithm nodes in the target combination algorithm.

[0153] In some embodiments of this application, a target visual recognition algorithm is generated based on a target combination algorithm, a visual recognition scene, and visual recognition warning conditions, including:

[0154] S701. In the preset scene configuration mapping table, determine the image enhancement strategy and result filtering strategy for visual recognition scene matching.

[0155] In the embodiments of this application, the preset scene configuration mapping table refers to a pre-built structured data index library that stores the correspondence between visual environment feature labels and algorithm processing strategies, such as a key-value database or an Extensible Markup Language (XML) configuration file. Image enhancement strategies refer to a set of image signal processing parameters or algorithm model selection (such as dehazing, noise reduction, and wide dynamic range processing) for specific ambient lighting, weather, or sensor noise problems. Result filtering strategies refer to a set of rules for confidence threshold truncation, geometric constraint filtering, or temporal smoothing processing of the model inference output results.

[0156] In some embodiments of this application, the determination process employs a semantic tag-based lookup table retrieval mechanism. The system uses the visual recognition scene identified in the preceding steps (e.g., "low illumination at night," "dense fog," or "strong backlight environment") as the query key to search a preset scene configuration mapping table. If a record is found that matches the semantic match, the system reads the corresponding image enhancement strategy (e.g., "enable gamma correction" or "load low-light enhancement network") and result filtering strategy (e.g., "reduce detection confidence threshold" or "enable Kalman filter-based trajectory smoothing"). This step ensures that the generated algorithm can adaptively adjust to specific environmental challenges.

[0157] S702, The image enhancement strategy is determined as a visual preprocessing algorithm, and the result filtering strategy is determined as a visual postprocessing algorithm.

[0158] In the embodiments of this application, the visual preprocessing algorithm refers to a computational module that standardizes, enhances, or repairs image pixel data before inputting image data into a neural network model. The visual postprocessing algorithm refers to a computational module that decodes, deduplicates, and cleans business logic of detection boxes, class probabilities, or segmentation masks after the neural network model outputs the original tensors.

[0159] In some embodiments of this application, the determination process employs component instantiation technology. Based on the image enhancement strategy determined in S701, the system calls the corresponding image processing function or lightweight neural network model from a pre-set algorithm operator library and initializes it using the parameters in the strategy, thereby instantiating an executable visual preprocessing algorithm object. Similarly, based on the result filtering strategy, the system instantiates a visual post-processing algorithm object containing non-maximum suppression logic or geometric rule filtering logic. For example, for a "rainy day" scene, the system loads a rain removal model based on a generative adversarial network (GAN) as the visual preprocessing algorithm to eliminate the interference of rain streaks on recognition accuracy.

[0160] S703, Generate Algorithm Execution Pipeline, wherein, in the algorithm execution pipeline, the data output port of the visual preprocessing algorithm is connected to the data input port of the target combination algorithm, and the data output port of the target combination algorithm is connected to the data input port of the visual postprocessing algorithm.

[0161] In the embodiments of this application, the algorithm execution pipeline refers to a data stream processing architecture that serializes and connects multiple independent computing units. Data output ports and data input ports refer to memory interfaces or video memory pointers used between algorithm modules to transfer image frame data, feature tensors, or metadata.

[0162] In some embodiments of this application, the generation process constructs a directed graph based on a streaming media data processing framework. The system sequentially creates visual preprocessing algorithm nodes, target combination algorithm nodes, and visual post-processing algorithm nodes, and defines a buffer transfer mechanism between nodes. Specifically, the system configures the visual preprocessing algorithm to write its processed image frames into a shared video memory area, and passes a pointer to this video memory area to the input port of the target combination algorithm. Subsequently, the system passes the raw detection results inferred by the target combination algorithm to the visual post-processing algorithm. By establishing this tightly coupled data path, efficient end-to-end processing from the raw video stream to the structured recognition results is achieved, and the latency caused by data copying is effectively reduced.

[0163] S704. Configure the visual recognition warning conditions into the logical decision layer of the algorithm execution pipeline to obtain the target visual recognition algorithm.

[0164] In the embodiments of this application, the logic decision layer refers to the decision module located at the end of the algorithm execution pipeline, which is used to interpret the structured recognition results and trigger specific events according to business rules.

[0165] In some embodiments of this application, the configuration process includes rule parsing and code injection. The system first performs semantic parsing on visual recognition warning conditions in natural language form (e.g., "a person was detected staying in a restricted area for more than 10 seconds"), converting them into computer-executable logical expressions or scripts. For warning conditions involving the time dimension, a finite state machine or time window counter is constructed within the logical decision layer to track the state changes of the target object in consecutive frames. Subsequently, the system injects this logical expression into the end node of the algorithm execution pipeline. Finally, the entire pipeline, including the constructed visual preprocessing algorithm, target combination algorithm, visual post-processing algorithm, and the configured logical decision layer, is encapsulated into a single deployable package, which is the target visual recognition algorithm.

[0166] As can be seen, the embodiments of this application determine the visual preprocessing algorithm and the visual postprocessing algorithm based on the visual recognition scenario, and connect them with the target combination algorithm in the algorithm execution pipeline, configure the visual recognition warning conditions to the logic decision layer, and generate an algorithm execution pipeline with full-process image processing and business logic judgment capabilities.

[0167] In some embodiments of this application, after generating the algorithm execution pipeline, the following is also included:

[0168] S801. Calculate the total resource consumption value of the algorithm based on the resource consumption attributes of each algorithm node in the algorithm execution pipeline.

[0169] In the embodiments of this application, resource consumption attributes refer to quantitative indicators describing the hardware resources occupied by each algorithm node during runtime, including floating-point operations per second, video memory usage, and memory bandwidth requirements. The total resource consumption value of the algorithm refers to the cumulative resource requirements required for the algorithm execution pipeline to complete one inference process on the target camera device.

[0170] In some embodiments of this application, the determination process employs a static graph analysis method. The system parses the directed acyclic graph structure of the algorithm execution pipeline, traversing all nodes, including the visual preprocessing algorithm, the target combination algorithm, and the visual postprocessing algorithm. For serially connected nodes, the system accumulates their resource consumption attributes; for parallel branch nodes (e.g., multiple detection subtasks running simultaneously), the system calculates the peak resource consumption of each branch within the same time slice to ensure that the evaluation results cover the worst-case load requirements. The final value obtained is the total resource consumption value of the algorithm.

[0171] S802. Obtain the current available resource threshold of the target camera device.

[0172] In the embodiments of this application, the current available resource threshold refers to the maximum computing power that the target camera device can allocate to the newly added visual recognition algorithm, provided that the operating system kernel, video encoding and streaming service and network protocol stack and other resident processes are running stably.

[0173] In some embodiments of this application, the acquisition process is implemented through a device status monitoring interface. The system queries the hardware specifications of the target camera device (such as total computing power and total video memory) and the current system load rate in real time. Subsequently, the system deducts a portion of the idle resources as a buffer according to a preset security redundancy strategy, and the remaining portion is confirmed as the current available resource threshold. For example, if the device's total video memory is 4 gigabytes (GB), 1GB is currently occupied, and the system reserves 0.5GB as a buffer, then the current available resource threshold is 2.5GB.

[0174] S803. If the total resource consumption of the algorithm is greater than the current available resource threshold, then the algorithm execution pipeline will be pruned based on the visual recognition scenario.

[0175] In the embodiments of this application, algorithm node pruning refers to the process of removing unnecessary, redundant, or low-contribution algorithm nodes from the pipeline based on scenario characteristics when it is identified that the current hardware resources are insufficient to support the complete algorithm execution pipeline, in order to reduce the overall computing load.

[0176] In some embodiments of this application, this step includes scene correlation analysis and topology reconstruction. The system first analyzes the current visual recognition scene (e.g., "indoor conference room" or "nighttime infrared mode"). Next, the system checks the correlation score between the functional definition of each node in the algorithm execution pipeline and the current scene. For example, if the scene is "indoor conference room" and the lighting conditions are stable, but the pipeline contains "defogging algorithm nodes" or "strong light suppression algorithm nodes" for dealing with severe outdoor weather, the system determines that the correlation scores of these nodes are below a preset retention threshold.

[0177] In some embodiments of this application, the specific operation of pruning is as follows: the algorithm node to be pruned is removed from the pipeline topology, and a bypass connection is established, that is, the output port of the upstream node of the node is directly connected to the input port of the downstream node of the node, so that the data flow skips the computing unit. By pruning low-priority visual preprocessing algorithms or simplifying visual postprocessing algorithms step by step, until the total resource consumption value of the updated algorithm is less than or equal to the current available resource threshold.

[0178] S804. If the total resource consumption of the algorithm is greater than the current available resource threshold and the pruning condition cannot be met, then the computation of the algorithm execution pipeline will be unloaded based on the collaborative computing strategy.

[0179] In the embodiments of this application, the inability to meet the pruning condition means that the remaining nodes in the algorithm execution pipeline are all core business nodes (e.g., core object detection models), and removing any one of these nodes will cause the task to fail. Computation offloading refers to migrating computationally intensive algorithm nodes to upper-level devices with more abundant computing power (such as edge servers or cloud platforms) for execution.

[0180] In some embodiments of this application, the system employs a graph segmentation algorithm to divide the algorithm execution pipeline. The system seeks the optimal segmentation point, minimizing the data transmission latency (transmission of intermediate feature maps or regions of interest images cropped from the target camera device) at the segmentation point, while ensuring that both the algorithm nodes (typically lightweight preprocessing or initial screening networks) retained on the target camera device (edge-side) and the algorithm nodes migrated to the cloud (typically high-precision target combination algorithms) satisfy the resource constraints of their respective host devices. For example, the system retains the "visual preprocessing algorithm" and lightweight "motion detection" for execution on the target camera device. When a moving target is detected, its image region is sent to the cloud server via the network, where the cloud server executes the complex "target combination algorithm" and "visual post-processing algorithm," finally returning the recognition result. The system automatically inserts a serialization transmission module at the segmentation point, reconstructing the algorithm execution pipeline into an edge-cloud collaborative architecture.

[0181] As can be seen, in this embodiment of the application, when the total resource consumption of the algorithm is greater than the current available resource threshold of the target camera device, the algorithm execution pipeline is pruned based on the visual recognition scene so that the generated algorithm execution pipeline can be adapted to the current actual available resources of the target camera device.

[0182] In some embodiments of this application, algorithm node pruning is performed on the algorithm execution pipeline based on the visual recognition scenario, including:

[0183] S901. Obtain environmental attribute parameters of the visual recognition scene.

[0184] In the embodiments of this application, the environmental attribute parameters are a set of key-value pairs describing the physical characteristics of the scene. These parameters originate from the parsing results of the natural language description input by the user in step S102, or from statistical analysis of historical data collected by the target camera device. For example, if the user inputs the natural language command "monitor the entrance to the B3 level of the underground parking lot," the following environmental attribute parameters can be obtained after parsing by the large language model and combining it with the camera's historical metadata:

[0185] Illumination: Constant Low-Light;

[0186] Weather: No weather change indoors (Indoor / None);

[0187] Background Complexity: Static background;

[0188] Target Velocity: Slow (vehicle speed in underground parking garage <20 km / h).

[0189] S902. Traverse each functional node in the algorithm execution pipeline and obtain the activation condition of each functional node.

[0190] In the embodiments of this application, each algorithm module (i.e., functional node) in the algorithm execution pipeline defines in its metadata the preconditions for it to perform as expected or for it to exist, i.e., activation conditions. Activation conditions include the core problem domain solved by the node and the environmental thresholds that take effect, for example:

[0191] The activation condition for function node A, “De-rain / snow algorithm”, is: Weather == Rain OR Weather == Snow.

[0192] The activation condition for functional node B, "Wide Dynamic Range (WDR)", is: IlluminationType == HighContrast (high contrast lighting, such as at the tunnel entrance).

[0193] The activation condition for function node C, "Inter-frame differential motion detection algorithm", is: CameraStatus == Fixed and Background == Static.

[0194] The activation condition for the functional node D "Heavy Vehicle Multi-Feature Tracking Algorithm (DeepSORT)" is: TargetDensity == High (dense target) or Occlusion == Frequent (frequent occlusion).

[0195] S903. Calculate the matching score between environmental attribute parameters and activation conditions.

[0196] In the embodiments of this application, the system logically compares the actual scene parameters obtained in S901 with the theoretical activation conditions obtained in S902 to obtain a matching score between the environmental attribute parameters and the activation conditions. The matching score can be, for example, a floating-point number between 0 and 1.

[0197] Hardly mutually exclusive scenarios: If there is a logical conflict between the environmental attribute parameters and the activation conditions, the matching score is 0. For example, for the "rain and snow removal algorithm", when the environmental attribute parameter of the scene is Weather: Indoor / None (indoor), the rain removal function is not needed, and the matching score is 0.

[0198] Partial Matching Scenarios: If some environmental attribute parameters meet the conditions, a score is calculated based on preset weights. For example, the "heavy vehicle multi-feature tracking algorithm" is mainly used to solve complex occlusion problems. If the environmental attribute parameters of the current scene are Target Velocity: Slow and TargetDensity: Low (single lane in underground parking garage, few cars and slow speed), a simple Kalman filter can meet the requirements. Therefore, the system determines that its necessity is low, and the matching score is 0.3.

[0199] A perfect match scenario: If the environmental attribute parameters completely meet the activation conditions, such as the scenario being "night" and the node being "low light enhancement", then the matching score is 1.

[0200] S904. If the matching score is lower than the preset necessity threshold, the corresponding functional node will be removed from the algorithm execution pipeline and the data connection will be re-established.

[0201] In the embodiments of this application, re-establishing a data connection means that if a node is deleted (e.g., node B is deleted), the output port of the upstream node A of node B is directly remapped to the input port of the downstream node C of node B to ensure the continuity of the data flow.

[0202] In some embodiments of this application, the necessity threshold may be, for example, 0.5. For example, in step S903:

[0203] For the "rain and snow removal algorithm" (score 0 < 0.5): the system removes it from the pipeline.

[0204] For the "heavy vehicle multi-feature tracking algorithm" (score 0.3 < 0.5): the system removes it and replaces it with a "distance-based simple matching algorithm (e.g., cross-union matcher)" which consumes less computing power.

[0205] As can be seen, this application embodiment introduces an "environmental attribute-activation condition" matching mechanism to deeply analyze the underlying logic such as "no need for rain removal algorithm when there is no rain indoors" and "no need for complex tracking on one-way streets," thereby reducing the computing power requirements of edge devices without sacrificing the accuracy required by the business.

[0206] In some embodiments of this application, after generating the target visual recognition algorithm, the method further includes:

[0207] S1001. Utilize a large language model to analyze the semantic boundaries of the visual recognition task and generate positive sample descriptions and counterfactual negative sample descriptions.

[0208] In the embodiments of this application, semantic boundaries refer to the range of judgment of visual concepts in the feature space. A positive sample description is a textual description of a typical scene that conforms to the definition of a visual recognition task. For example, if the task is "detecting smoking," the positive sample description generated would be "A worker is holding a lit cigarette, and smoke is emanating from it."

[0209] In some embodiments of this application, counterfactual negative sample descriptions are scene descriptions that are visually similar to the target object but semantically do not belong to the target object. The large language model utilizes its built-in common sense base to retrieve easily confused difficult examples through counterfactual reasoning. Continuing with the "detecting smoking" example above, the counterfactual negative sample descriptions generated by the large language model include: "A worker has a lollipop in his mouth," "A worker exhales white breath in cold weather," and "A worker holds chalk and writes."

[0210] S1002. Using an image generation model, based on positive sample descriptions and counterfactual negative sample descriptions, generate an adversarial test image set containing realistic backgrounds.

[0211] In the embodiments of this application, the image generation model is a deep learning model with text-to-image generation capabilities, such as the Latent Diffusion Model (LDM). The adversarial test image set is a virtual dataset with automatically labeled information. Since each image is generated from an explicit text description, the system automatically labels images generated from positive sample descriptions as "Positive" and images generated from counterfactual negative sample descriptions as "Negative," thereby constructing a "Ground Truth" for verification without human intervention.

[0212] In some embodiments of this application, the adversarial test image set containing a realistic background refers to images in the adversarial test image set that are not isolated target objects, but rather incorporate environmental features of the visual recognition scene determined in previous steps. For example, the aforementioned "worker with a lollipop" is incorporated into the background of a "factory workshop" instead of a blank background to simulate real surveillance interference.

[0213] S1003. Conduct virtual drills of the target visual recognition algorithm on an adversarial test image set and calculate the false alarm rate.

[0214] In the embodiments of this application, virtual training refers to the process of using the synthetic images generated in S1002 as input streams and feeding them in batches into the generated target visual recognition algorithm for inference and prediction. The false alarm rate refers to the proportion of all counterfactual negative samples labeled "negative" that are incorrectly identified as "having a target object" by the target visual recognition algorithm, reflecting the algorithm's ability to resist interference in easily confused scenarios. For example, if 20 out of 100 images of "holding a lollipop" and "exhaling white breath" are identified as "smoking" by the algorithm, the false alarm rate is 20%.

[0215] S1004. If the false alarm rate is higher than the preset threshold, the large language model is used to negatively constrain the visual cue words in the target visual recognition algorithm based on the characteristics of counterfactual negative samples until the exercise passes.

[0216] In the embodiments of this application, the visual cue words are the original text of the text vector configured in the target underlying algorithm for feature matching. Negative constraint correction refers to adding exclusionary descriptions to the visual cue words.

[0217] In some embodiments of this application, when the false positive rate exceeds the threshold, a large language model analyzes the visual features (e.g., "stick-shaped object," "white smoke-like object") of specific counterfactual negative samples that cause false positives. Subsequently, the input prompts for the target visual recognition algorithm are modified. For example, the initial prompt "cigarette" is modified to "cigarette, excluding lollipops, excluding chalk, excluding cold breath." This modified prompt is re-encoded and updated in the algorithm model, forming a new discrimination boundary. The system repeatedly executes the above generation and testing steps until the false positive rate falls below a preset threshold (e.g., 5%). At this point, the algorithm is considered to have passed the rehearsal, ensuring reliability in the absence of real samples.

[0218] As can be seen, the embodiments of this application utilize the counterfactual reasoning capability of the large language model and the synthesis capability of the image generation model to perform adversarial attacks and targeted hardening on the algorithm, so that the generated algorithm can complete adaptive optimization for various interference scenarios before deployment, thereby reducing the risk of false alarms in actual application scenarios.

[0219] In some embodiments of this application, after generating the target visual recognition algorithm and deploying the target visual recognition algorithm to the target camera device, the method further includes an online dynamic adaptive step for the target visual recognition algorithm:

[0220] S1101, Online monitoring phase: Real-time acquisition of multi-dimensional performance data, environmental perception data, and equipment resource status data generated by the target camera device when running the target visual recognition algorithm.

[0221] In the embodiments of this application, multidimensional performance data is a set of quantitative indicators used to characterize the quality of the current algorithm's operation. Specifically, the confidence distribution output by the target visual recognition algorithm refers to a statistical distribution histogram composed of the probability scores of all detected targets within a time window (e.g., the past 30 seconds). If the distribution center shifts towards low-score regions even in normal scenes, it indicates that the algorithm's performance may be declining. Target detection rate fluctuation refers to the magnitude of change in the number of detected targets per unit time relative to a historical baseline. The entropy value of the activation map of the algorithm's internal feature layers is a measure of the degree of information disorder in the feature maps of the intermediate layers of the deep neural network; drastic changes in entropy value usually indicate a deviation between the distribution of input data and model training data.

[0222] Environmental perception data describes the physical environment characteristics of the video stream itself. It is acquired without relying on complex large models, but rather through analysis by an ultra-lightweight environmental perception network running on the target camera device with extremely low computational cost. Specifically, the dynamic illumination index reflects the temporal gradient of the overall brightness of the image and the presence of strong backlighting or flicker; the weather feature vector is a multi-dimensional value representing whether the current condition is rain, snow, fog, or sunshine; and the background perturbation coefficient quantifies the degree of dynamic change in the background region (non-interest area), such as swaying leaves or water ripples.

[0223] Device resource status data reflects the hardware health of the target camera device. Among them, real-time available computing power refers to the number of floating-point operations per second currently idle; memory utilization rate refers to the percentage of random access memory and video memory used; temperature status refers to the real-time junction temperature of the core processor. Excessive temperature will trigger hardware frequency reduction protection, thereby affecting the algorithm inference speed.

[0224] S1102, Performance Evaluation and Attribution Stage: Based on multi-dimensional performance data, calculate the current algorithm performance degradation index; when the performance degradation index exceeds the first preset threshold, use an online updated performance digital twin model to perform attribution analysis on the causes of degradation and output the attribution vector.

[0225] In the embodiments of this application, the performance degradation index is a comprehensive score calculated by weighting multi-dimensional performance data. When the score exceeds a first preset threshold, it indicates that the algorithm can no longer meet the expected Quality of Service (QoS) standard.

[0226] In embodiments of this application, the performance digital twin model is a lightweight regression prediction model (e.g., XGBoost (eXtreme Gradient Boosting, an optimized distributed gradient boosting library)) running locally or at an edge gateway. This model is continuously trained online using historical running data, fitting a mapping relationship between "data-performance".

[0227] In the embodiments of this application, the attribution vector is the output of the model, which is a probability vector, with each dimension corresponding to a different cause of failure. Among them, instruction semantic drift refers to the conceptual evolution of the target object in the real scene (for example, "inspection worker" originally meant wearing blue clothes, but now it means wearing yellow clothes), causing the initial natural language instruction to be inaccurate; scene condition abrupt change refers to a rapid change in the physical environment (such as a sudden rainstorm); resource insufficiency fluctuation refers to insufficient computing power due to other processes preempting it; target data distribution offset refers to the inconsistency between the statistical characteristics of the input image and the training data (domain offset).

[0228] S1103, Adaptive Decision and Execution Phase: Based on attribution vectors and device resource status data, a meta-controller deployed on the target camera device side dynamically selects and executes at least one adaptive action.

[0229] In the embodiments of this application, the meta-controller is a control process with high-level decision-making authority that is independent of the main visual recognition task. It selects a better strategy from a pre-defined adaptive action library based on the primary contradiction pointed to by the attribution vector.

[0230] In embodiments of this application, the adaptive action includes at least one of the following:

[0231] Action A (Algorithm Parameter Fine-Tuning Strategy): Applicable to situations where the attribution vector points to slight scene changes (e.g., decreased lighting). The meta-controller quickly restores performance by adjusting the detection confidence threshold (e.g., lowering the threshold to improve recall), the non-maximum suppression threshold, or the gain coefficient (Gamma value) of the image enhancement algorithm.

[0232] Action B (Dynamic Loading Strategy for Computational Modules): This strategy is suitable when the attribution vector points to resource fluctuations or specific scenario requirements, and when device resource status data allows it. For example, when computing power is detected to be strained, the high-precision ResNet-50 (a 50-layer version of a residual neural network) backbone network module is hot-switched to the lightweight ShuffleNet (a lightweight deep neural network architecture) module; this process utilizes dynamic link library loading technology and does not require stopping video stream processing.

[0233] Action C (Dynamic Pipeline Reorganization): Applicable when the attribution vector points to drastic environmental changes. For example, when the environmental perception network detects "rainy day", the meta-controller dynamically inserts a "rain removal preprocessing node" into the pipeline; when it becomes "sunny day", it is deleted to save computing power.

[0234] Action D (Trigger Semantic Clarification): Applicable when the attribution vector points to semantic drifting of the instruction and automatic adjustment fails. The meta-controller generates a minimal clarification question (e.g., "Most detected targets are in yellow uniforms, should the definition of 'worker' be updated to include yellow uniforms?") and pushes it to the user interface via a pop-up or message.

[0235] S1104, Federated Evolution Stage: The meta controller periodically uploads the performance change data and attribution vectors after performing adaptive actions locally to the cloud in encrypted form for federated aggregation; after aggregating the experience of multiple edge devices, the cloud generates a global evolution strategy and model increment, and distributes them to the meta controller.

[0236] In the embodiments of this application, the federated evolution stage aims to establish cross-device data interaction connections, leveraging swarm intelligence. Local devices only upload parameter update gradients and anonymized statistical information (performance change data and attribution vectors) to remove privacy data.

[0237] In the embodiments of this application, the global evolution strategy is a better adaptive rule base generated by the cloud server based on feedback from massive edge devices (e.g., if it is found that most devices need to reduce contrast in "dusk" scenes, this is solidified as a general rule). The model increment is a fine-tuning update package of the weights of the basic algorithm. After being distributed, the meta-controller uses this global knowledge to update its local decision logic, so as to realize common parameter updates and strategy optimization for multiple devices.

[0238] As can be seen, the embodiments of this application monitor and analyze by deploying a meta controller, thereby realizing the dynamic adjustment of algorithm parameters and structure to better cope with changes in the external environment.

[0239] Secondly, embodiments of this application provide an electronic device that integrates any of the natural language interaction-based visual recognition algorithm generation systems provided in the embodiments of this application. The electronic device includes a processor and a memory, the memory storing a computer program configured to be executed by the processor to implement the natural language interaction-based visual recognition algorithm generation method as described in any of the above embodiments.

[0240] Thirdly, embodiments of this application provide a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), a magnetic disk, or an optical disk, etc. The computer-readable storage medium stores a computer program configured to be executed by a processor to implement the visual recognition algorithm generation method based on natural language interaction as described in any of the preceding claims.

[0241] The embodiments of this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for generating visual recognition algorithms based on natural language interaction, characterized in that, The visual recognition algorithm generation method based on natural language interaction includes: Obtain input natural language interaction commands; Using a pre-defined large language model, the corresponding target camera device, visual recognition scene, visual recognition task, and visual recognition warning conditions are determined based on the natural language interaction instructions. The visual recognition task is broken down into multiple sub-tasks with logical dependencies; Determine the target base algorithm for matching each of the subtasks; The target base algorithms for matching multiple sub-tasks are all used as the target base algorithms for matching the visual recognition task. Based on the multiple target-based algorithms for matching the visual recognition task, the visual recognition scene, and the visual recognition warning conditions, a target visual recognition algorithm is generated, and the target visual recognition algorithm is applied to the target camera device.

2. The visual recognition algorithm generation method based on natural language interaction as described in claim 1, characterized in that, The target basic algorithm for determining the matching of each subtask includes: For each subtask, check in the preset basic algorithm database whether there is a preset basic algorithm that matches the subtask; If a preset basic algorithm for subtask matching exists in the basic algorithm database, then the target basic algorithm for subtask matching is determined based on the preset basic algorithm for subtask matching in the basic algorithm database. If the preset basic algorithm for matching the subtask does not exist in the basic algorithm database, a target basic algorithm for matching the subtask is generated based on the zero-sample mechanism.

3. The visual recognition algorithm generation method based on natural language interaction as described in claim 2, characterized in that, The step of determining the target basic algorithm for subtask matching based on the preset basic algorithm for subtask matching in the basic algorithm database includes: If the number of preset basic algorithms for subtask matching in the basic algorithm database is one, then the preset basic algorithm for subtask matching in the basic algorithm database shall be used as the target basic algorithm for subtask matching. If there are multiple preset basic algorithms for subtask matching in the basic algorithm database, then based on the computing resources of the target camera device, a corresponding target value for computing resource consumption is determined. Based on the target value for computing resource consumption, the target basic algorithm for subtask matching is determined from the multiple preset basic algorithms for subtask matching in the basic algorithm database.

4. The visual recognition algorithm generation method based on natural language interaction as described in claim 2, characterized in that, The underlying algorithm for generating the target matching for the subtask based on the zero-shot mechanism includes: Obtain the target object label corresponding to the subtask; Based on the target object label, generate visual cue description information; Using a preset text encoder, feature mapping is performed on the visual cue description information to obtain a text feature vector; The text feature vectors are configured into the feature matching layer of a pre-trained open vocabulary visual recognition model; The open vocabulary visual recognition model configured with the text feature vectors will be used as the target base algorithm for the subtask matching.

5. The visual recognition algorithm generation method based on natural language interaction as described in claim 1, characterized in that, The method for generating a target visual recognition algorithm based on multiple target-based algorithms for matching the visual recognition task, the visual recognition scene, and the visual recognition early warning conditions includes: Based on the logical dependency relationship, multiple target basic algorithms for matching the visual recognition task are combined to obtain a target combination algorithm; The target visual recognition algorithm is generated based on the target combination algorithm, the visual recognition scene, and the visual recognition warning conditions.

6. The visual recognition algorithm generation method based on natural language interaction as described in claim 5, characterized in that, The combination of multiple target-based algorithms for matching the visual recognition task based on the logical dependency relationship yields a target combination algorithm, including: Based on the aforementioned logical dependencies, the execution topology sequence of multiple target base algorithms for matching the visual recognition task is determined; Determine the output parameters of the target base algorithm located upstream in the execution topology sequence, and the input parameters of the target base algorithm located downstream; Establish a data mapping channel between the output parameters and the input parameters; The target combination algorithm is generated based on the execution topology sequence and the data mapping channel.

7. The visual recognition algorithm generation method based on natural language interaction as described in claim 5, characterized in that, The step of generating the target visual recognition algorithm based on the target combination algorithm, the visual recognition scene, and the visual recognition warning conditions includes: In the preset scene configuration mapping table, the image enhancement strategy and result filtering strategy for matching the visual recognition scene are determined; The image enhancement strategy is determined as a visual preprocessing algorithm, and the result filtering strategy is determined as a visual postprocessing algorithm. A generation algorithm execution pipeline is provided, wherein, in the algorithm execution pipeline, the data output port of the visual preprocessing algorithm is connected to the data input port of the target combination algorithm, and the data output port of the target combination algorithm is connected to the data input port of the visual postprocessing algorithm; The visual recognition warning conditions are configured into the logical decision layer of the algorithm execution pipeline to obtain the target visual recognition algorithm.

8. The visual recognition algorithm generation method based on natural language interaction as described in claim 7, characterized in that, Following the execution pipeline of the generation algorithm, the following also includes: Based on the resource consumption attributes of each algorithm node in the algorithm execution pipeline, the total resource consumption value of the algorithm is determined; Obtain the current available resource threshold of the target camera device; If the total resource consumption of the algorithm is greater than the current available resource threshold, then the algorithm execution pipeline is pruned based on the visual recognition scenario.

9. An electronic device, characterized in that, The electronic device includes a processor and a memory, the memory storing a computer program configured to be executed by the processor to implement the visual recognition algorithm generation method based on natural language interaction as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program configured to be executed by a processor to implement the visual recognition algorithm generation method based on natural language interaction as described in any one of claims 1 to 8.

Images