Target detection method, system and controller

Abstract

The present invention discloses a target detection method, system and controller, the method comprising: receiving an original image and a task prompt; performing semantic analysis on the task prompt to generate a task flow command; if the task command is an image processing command, calling a target image processing algorithm from a pre-built image algorithm library according to the image processing command, performing image processing on the original image using the target image processing algorithm, and generating an image processing result; if the task command is a detection command, calling a pre-trained target detection model from a pre-built model library according to the detection command, performing detection processing on the original image or the original image after image processing according to the target object using the pre-trained target detection model, and generating a detection result; obtaining a target detection reasoning result according to the image processing result and / or detection result generated based on each task command in the task flow command. The method makes the detection reasoning process explainable and adaptable.

Description

Technical Field

[0001] The present invention relates to the field of target detection technology, and in particular to a target detection method, system and controller. Background Art

[0002] In complex scenarios such as substations, the monitoring of safety risks is particularly important. The requirements for simultaneous detection of dozens of defects in substation scenarios are challenging for relevant detection systems, and relevant detection systems usually lack the interpretability and adaptability of the detection process for various defects. Summary of the invention

[0003] The present invention aims to solve one of the technical problems in the related art at least to a certain extent. To this end, one purpose of the present invention is to propose a target detection method that allows users to interact with the detection process, making the detection reasoning process interpretable and adaptable while also improving the efficiency of model detection.

[0004] The second objective of the present invention is to provide a target detection system.

[0005] The third object of the present invention is to provide a controller.

[0006] To achieve the above-mentioned purpose, the first aspect of the present invention proposes a target detection method, which includes: receiving an original image and a task prompt; performing semantic analysis on the task prompt to generate a task flow command, wherein the task flow command includes at least one task command; if the task command is an image processing command, then according to the image processing command, calling a target image processing algorithm from a pre-built image algorithm library, using the target image processing algorithm to perform image processing on the original image, and generating an image processing result; if the task command is a detection command, then according to the detection command, calling a pre-trained target detection model from a pre-built model library, using the pre-trained target detection model to perform detection processing on the original image or the original image after image processing according to the target object, and generating a detection result; according to the image processing result and / or the detection result generated based on each of the task commands in the task flow command, obtaining a target detection inference result.

[0007] According to the target detection method of the embodiment of the present invention, the original image and the task prompt are combined, a semantic analysis is performed on the task prompt, and a task flow command including at least one task command is generated. The original image is processed in sequence according to the task commands in the task flow command, so that the user can interact with the reasoning process, making the reasoning process explainable and adaptable, while also improving the efficiency of model detection.

[0008] In addition, the target detection method proposed in the above embodiment of the present invention may also have the following additional technical features:

[0009] According to one embodiment of the present invention, the method further includes: if the task command is a training command, then according to the training command, calling the target detection model to be trained from a pre-built model library, and training the target detection model to be trained using the original image or the original image after image processing to obtain a training result; if the task command is a test command, then according to the test command, calling the target detection model to be tested from a pre-built model library, and testing the target detection model to be tested using the original image or the original image after image processing to obtain a test result.

[0010] According to one embodiment of the present invention, generating the detection result includes: decoding the original result output by the pre-trained target detection model to obtain a decoded result; post-processing the decoded result, wherein the post-processing includes at least one of non-maximum suppression, inverse normalization, de-filling, and false detection and missed detection matching processing.

[0011] According to one embodiment of the present invention, the method further includes: receiving information fed back by a user based on the detection result; and adjusting the target image processing algorithm and / or the pre-trained target detection model based on the feedback information.

[0012] According to one embodiment of the present invention, the task command includes at least one of a target task type, a target image processing algorithm, a target detection model, a target image and a target object, wherein the target task type includes at least one of an image processing type, a detection type, a training type and a test type; the pre-built model library includes at least one of a yolov8 model, a yolov9 model, a BD_Net model and a gdino model; the pre-built image algorithm library includes at least one of an image enhancement algorithm, an image edge detection algorithm, an image segmentation algorithm, an image amplification algorithm and the like.

[0013] According to one embodiment of the present invention, the BD_Net model includes a backbone network and auxiliary branches; the backbone network includes a first input layer, a first convolutional layer, a second convolutional layer, a first feature extraction module, a third convolutional layer, a first attention module, a deformable convolutional layer, a second feature extraction module, a fourth convolutional layer, a second attention module, a third feature extraction module, a fifth convolutional layer, a third attention module, and a fourth feature extraction module connected in sequence; the second feature extraction module is connected to the first head in sequence through a first attention mechanism, a first splicing layer, a second attention mechanism, and a fifth feature extraction module; the third feature extraction module is connected to the first head in sequence through a third attention mechanism, a second splicing layer, a fourth attention mechanism, and a sixth feature extraction module. The module is connected to the second head; the fourth feature extraction module is also connected to the third head through the fifth attention mechanism, the pooling layer, the sixth attention mechanism, and the seventh feature extraction module in sequence; wherein the pooling layer is connected to the second splicing layer through the first upsampling, the second splicing layer is connected to the first splicing layer through the eighth feature extraction module and the second upsampling, the fifth feature extraction module is connected to the sixth feature extraction module through the sixth convolution layer and the first feature fusion module, the sixth feature extraction module is connected to the seventh feature extraction module through the seventh convolution layer and the second feature fusion module, the third attention mechanism is connected to the first feature fusion module, and the fifth attention mechanism is connected to the second feature fusion module;The auxiliary branch includes a second input layer, a seventh convolution layer, an eighth convolution layer, a ninth feature extraction module, an attention and deformable convolution module, a ninth convolution layer, a first fusion layer, a tenth feature extraction module, a tenth convolution layer, a second fusion layer, an eleventh feature extraction module, an eleventh convolution layer, a third fusion layer, a twelfth feature extraction module, a twelfth convolution layer, a third feature fusion module, a thirteenth feature extraction module, a thirteenth convolution layer, a fourth feature fusion module and a fourteenth feature extraction module, the third fusion layer is connected to the fourth feature fusion module, the twelfth feature extraction module is connected to the fourth head, the thirteenth feature extraction module is connected to the fifth head, and the fourteenth feature extraction module is connected to the sixth head. The auxiliary branch also includes a seventh attention mechanism, a first fusion layer, a second fusion layer, a tenth feature extraction module, a tenth convolution layer, a second fusion layer, a twelfth feature extraction module, a twelfth convolution layer, a third feature fusion module, a thirteenth feature extraction module, a thirteenth convolution layer, a fourth feature fusion module and a fourteenth feature extraction module. The third fusion layer is connected to the fourth feature fusion module, the twelfth feature extraction module is connected to the fourth head, the thirteenth feature extraction module is connected to the fifth head, and the fourteenth feature extraction module is connected to the sixth head. The eighth attention mechanism and the ninth attention mechanism, the seventh attention mechanism, the eighth attention mechanism and the ninth attention mechanism are also connected to the second fusion layer, the seventh attention mechanism, the eighth attention mechanism and the ninth attention mechanism are also connected to the input end of the fourth feature fusion module, the eighth attention mechanism is also connected to the first fusion layer, the seventh attention mechanism is also connected to the third fusion layer, the seventh attention mechanism is also connected to the third feature fusion module, wherein the second feature extraction module in the backbone network is connected to the seventh attention mechanism, the third feature extraction module in the backbone network is connected to the eighth attention mechanism, and the fourth feature extraction module in the backbone network is connected to the ninth attention mechanism. ;

[0014] According to one embodiment of the present invention, the feature extraction module includes four DS_GELAN modules connected once, and the DS_GELAN module includes: a third input layer, a fourteenth convolutional layer, a fifteenth convolutional layer, a first cascade submodule, a second cascade submodule, a third cascade submodule and a third splicing layer, the output end of the third input layer is respectively connected to the input ends of the fourteenth convolutional layer and the fifteenth convolutional layer, the output end of the fourteenth convolutional layer is connected to the input end of the third splicing layer, the output end of the fifteenth convolutional layer is connected to the input end of the first cascade submodule, and the output end of the first cascade submodule is connected to the input end of the third splicing layer. The output end of the first cascade submodule is connected to the input end of the second cascade submodule, the output end of the second cascade submodule is connected to the input end of the third splicing layer, the output end of the second cascade submodule is connected to the input end of the third cascade submodule, and the output end of the third cascade submodule is connected to the input end of the third splicing layer, wherein the first cascade submodule includes a sixteenth convolutional layer, a first depth-separable convolutional layer, a second depth-separable convolutional layer, a seventeenth convolutional layer and a first addition layer, the input ends of the sixteenth convolutional layer and the first depth-separable convolutional layer are connected to the input end of the first cascade submodule, and the tenth The output end of the sixth convolutional layer is connected to the input end of the first addition layer, the output end of the first depth-separable convolutional layer is connected to the input end of the second depth-separable convolutional layer, the output end of the second depth-separable convolutional layer is connected to the input end of the seventeenth convolutional layer, the output end of the seventeenth convolutional layer is connected to the input end of the first addition layer, and the output end of the first addition layer is connected to the output end of the first cascade submodule; the second cascade submodule includes an eighteenth convolutional layer, a nineteenth convolutional layer, a third depth-separable convolutional layer, a fourth depth-separable convolutional layer, a twentieth convolutional layer and a second addition layer, and the input end of the eighteenth convolutional layer is connected to the output end of the first cascade submodule. The input end of the second cascade sub-module is connected, the output end of the eighteenth convolutional layer is connected to the input end of the nineteenth convolutional layer, the output end of the nineteenth convolutional layer is connected to the input end of the second addition layer, the output end of the eighteenth convolutional layer is connected to the input end of the third depthwise separable convolutional layer, the output end of the third depthwise separable convolutional layer is connected to the input end of the fourth depthwise separable convolutional layer, the output end of the fourth depthwise separable convolutional layer is connected to the input end of the twentieth convolutional layer, the output end of the twentieth convolutional layer is connected to the input end of the second addition layer, and the output end of the second addition layer is connected to the output end of the second cascade sub-module;The third cascade submodule includes a twenty-first convolutional layer, a twenty-second convolutional layer, a fifth depthwise separable convolutional layer, a sixth depthwise separable convolutional layer, a twenty-third convolutional layer, a seventh depthwise separable convolutional layer, an eighth depthwise separable convolutional layer and a twenty-fourth convolutional layer, a fourth splicing layer and a third addition layer, the input ends of the twenty-first convolutional layer, the fifth depthwise separable convolutional layer and the seventh depthwise separable convolutional layer are connected to the input end of the third cascade submodule, the output end of the twenty-first convolutional layer is connected to the input end of the twenty-second convolutional layer, the output end of the twenty-second convolutional layer is connected to the input end of the third addition layer, and the output end of the fifth depthwise separable convolutional layer is connected to the input end of the The input end of the sixth depthwise separable convolutional layer is connected, the output end of the sixth depthwise separable convolutional layer is connected to the input end of the twenty-third convolutional layer, the output end of the twenty-third convolutional layer is connected to the input end of the fourth concatenation layer, the output end of the seventh depthwise separable convolutional layer is connected to the input end of the eighth depthwise separable convolutional layer, the output end of the eighth depthwise separable convolutional layer is connected to the input end of the twenty-fourth convolutional layer, the output end of the twenty-fourth convolutional layer is connected to the input end of the fourth concatenation layer, the output end of the fourth concatenation layer is connected to the input end of the third addition layer, and the output end of the third addition layer is connected to the output end of the third cascade submodule. ;

[0015] According to one embodiment of the present invention, the feature fusion module includes a twenty-fifth convolutional layer, a fifth splicing layer, a twenty-sixth convolutional layer, a twenty-seventh convolutional layer, a sixth splicing layer, a twenty-eighth convolutional layer and an output layer connected in sequence, the twenty-fifth convolutional layer is connected to the first input end and the second input end of the feature fusion module, the feature fusion module also includes a twenty-ninth convolutional layer, the input end of the twenty-ninth convolutional layer is connected to the third input end of the feature fusion module, the output end of the twenty-ninth convolutional layer is connected to the input end of the sixth splicing layer, and the output layer is connected to the output end of the feature fusion module.

[0016] To achieve the above-mentioned purpose, the second aspect of the present invention proposes a target detection system, which includes: a receiving module, which is used to receive an original image and a task prompt; a semantic parsing module, which is used to perform semantic analysis on the task prompt and generate a task flow command, wherein the task flow command includes at least one task command; a processing module, which is used to call a target image processing algorithm from a pre-built image algorithm library according to the image processing command when the task command is an image processing command, and use the target image processing algorithm to perform image processing on the original image to generate an image processing result; when the task command is a detection command, according to the detection command, call a pre-trained target detection model from a pre-built model library, and use the pre-trained target detection model to perform detection processing on the original image or the original image after image processing according to the target object to generate a detection result; an output module, which is used to obtain a target detection inference result according to the image processing result and / or the detection result generated based on each task command in the task flow command.

[0017] To achieve the above-mentioned purpose, the third aspect of the present invention proposes a controller, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the target detection method proposed in the first aspect of the present invention is implemented.

[0018] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a flow chart of a target detection method according to an embodiment of the present invention;

[0020] Figure 2 is a flowchart of target detection according to a specific embodiment of the present invention;

[0021] Figure 3 is a flow chart of generating a detection result according to an embodiment of the present invention;

[0022] Figure 4 is a flow chart of a target detection method according to another embodiment of the present invention;

[0023] Figure 5 It is a structural diagram of the BD_Net model backbone network of an embodiment of the present invention;

[0024] Figure 6 is a structural diagram of an auxiliary branch of a BD_Net model according to an embodiment of the present invention;

[0025] Figure 7is a structural diagram of a DS_GELAN module according to an embodiment of the present invention;

[0026] Figure 8 is a structural diagram of a feature fusion module according to an embodiment of the present invention;

[0027] Fig. 9 is a schematic diagram of a target detection system according to an embodiment of the present invention;

[0028] Fig.10 is a schematic diagram of a target detection system according to a specific embodiment of the present invention;

[0029] Fig.11 4 is a structural block diagram of a controller according to an embodiment of the present invention. DETAILED DESCRIPTION

[0030] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0031] The target detection method, system and controller of the embodiments of the present invention are described in detail below with reference to the accompanying drawings and specific implementation methods.

[0032] Figure 1 FIG. 1 is a flow chart of a target detection method according to an embodiment of the present invention. Figure 1 As shown, the target detection method may include:

[0033] S101, receiving an original image and a task prompt;

[0034] S102, performing semantic analysis on the task prompt to generate a task flow command, wherein the task flow command includes at least one task command;

[0035] S103, if the task command is an image processing command, then according to the image processing command, a target image processing algorithm is called from a pre-built image algorithm library, and the original image is processed using the target image processing algorithm to generate an image processing result;

[0036] S104, if the task command is a detection command, then according to the detection command, a pre-trained target detection model is called from a pre-built model library, and the pre-trained target detection model is used to perform detection processing on the original image or the original image after image processing according to the target object to generate a detection result;

[0037] S105, obtaining a target detection inference result according to the image processing result and / or detection result generated based on each task command in the task flow command.

[0038] In order to solve the lack of interpretability and adaptability of various defect detections in the target detection process, the embodiment of the present invention receives the original image and the task prompt, and detects the original image according to the task prompt. Specifically, a large language model is used to perform semantic analysis on the task prompt, and a task flow command including at least one task command is generated. The original image is processed in sequence according to the task commands in the task flow command, and the task processing results corresponding to each task command are generated, so that each processing process of the target detection is visualized, and the interpretability and adaptability of the target detection process are achieved.

[0039] The embodiment of the present invention pre-builds and maintains an image algorithm library including various image processing and a model library including various models.

[0040] It is feasible to provide a natural language interface while providing an image input interface to receive the original image and task prompts input by the user. It should be noted that the original image received by the embodiment of the present invention may be one or more. The multiple images in the embodiment of the present invention are greater than or equal to two. The embodiment of the present invention does not limit the number of original images.

[0041] A large language model is used to perform semantic analysis on the task prompts to generate a task flow command including at least one task command. The task command type is determined according to the task command, such as an image processing command, a detection command, a training command, a test command, etc., so as to generate a corresponding Python program to execute the command according to the corresponding command type.

[0042] It is feasible to use the GPT-3 (Generative Pre-trained Transformer 3) model to perform semantic analysis on the task prompts. The GPT-3 model extracts the parameter names in the task prompts (such as "train", "detect", "image", "object", etc.) and determines the value "field" corresponding to each parameter name (for example, there will be a dictionary dict like this in the source code: {"train": Train, "detect": Det, "test": Test}, where Train, Det, Test are the values ​​corresponding to each parameter). The GPT-3 model maps the parameter names in the task prompts with the preset parameters to ensure that the parameter names in the task prompts can correctly correspond to the corresponding preset parameters, and generates the corresponding task commands based on the parameter names mapped to the preset parameters. At the same time, the GPT-3 model can generate corresponding python programs based on key parameters.

[0043] It should be noted that the embodiments of the present invention do not limit the specific large language model used. The GPT-4 model can also be used.

[0044] According to the task command, call the relevant algorithm or model to execute the task command. Specifically, according to the image processing algorithm name (such as "Crop", "Noise", "Fuzzification") and / or the detection model name (such as "yolov9", "gdino", "External", etc.) in the task command, match it with the image algorithm in the image algorithm library or the model in the model library, dynamically combine and call the corresponding image processing algorithm and / or defect detection model, and generate the corresponding image processing command, training command, detection command, test command, etc. According to the task flow command, build the corresponding reasoning process program, automatically configure the image processing steps, and load the trained model for real-time reasoning.

[0045] The embodiment of the present invention processes the original image in combination with the task prompt input by the user, so that the user can interact with the reasoning process, making the reasoning process explainable and adaptable while also improving the efficiency of model training, testing and detection.

[0046] In one embodiment of the present invention, the pre-built model library includes at least one of the yolov8 model, the yolov9 model, the BD_Net model, and the gdino model.

[0047] Specifically, the detection model module uses pre-trained YOLO series models, including but not limited to YOLOv8 model and YOLOv9 model.

[0048] Models such as BD_Net and gdino in the embodiments of the present invention can perform defect detection on the original image or the original image processed by the image algorithm and locate the defect target.

[0049] It should be noted that the models in the pre-built model library in the embodiment of the present invention are not limited to the above detection models, but also include other detection models. The embodiment of the present invention does not limit the models in the pre-built model library.

[0050] In one embodiment of the present invention, the pre-built image algorithm library includes at least one of an image enhancement algorithm, an image edge detection algorithm, an image segmentation algorithm, an image augmentation algorithm, and the like.

[0051] Specifically, image algorithms are used to process the original images, such as denoising, enhancement, and segmentation, to improve the accuracy of the detection model. Or data amplification is performed on the original images, such as rotation, cropping, color adjustment, adding noise, etc., so as to use the amplified original images to train the target detection model to be trained and enrich the training set.

[0052] It should be noted that the algorithms in the pre-built image algorithm library in the embodiment of the present invention are not limited to the above algorithms, but also include other algorithms. The embodiment of the present invention does not limit the algorithms in the pre-built image algorithm library.

[0053] In one embodiment of the present invention, the task command includes at least one of a target task type, a target image processing algorithm, a target detection model, a target image and a target object, wherein the target task type includes at least one of an image processing type, a detection type, a training type and a test type.

[0054] The embodiment of the present invention selects a model architecture, configures detection parameters, and generates a detection script according to the parameters in the task command.

[0055] Specifically, the type of the task command can be determined according to the target task type in the task command. If the target task type in the task command is an image processing type, the task command can be determined to be an image processing command. If the target task type in the task command is a detection type, the task command can be determined to be a detection command. If the target task type in the task command is a training type, the task command can be determined to be a training command. If the target task type in the task command is a test type, the task command can be determined to be a test command. Generate a corresponding python program according to the type of the task command and execute the task command.

[0056] When the task command is an image processing command, the specific algorithm used for image processing is determined according to the target image processing algorithm in the task command. The specific image to be processed and the specific detection target can be determined according to the target image and target object in the task command. Specifically, according to the image processing command, the target image processing algorithm is called from the pre-built image algorithm library, and the original image is processed using the target image processing algorithm to generate an image processing result.

[0057] When the task command is a detection command, the specific model for detection is determined according to the target detection model in the task command. The specific image to be processed and the specific detection target can be determined according to the target image and target object in the task command. Specifically, according to the detection command, a pre-trained target detection model is called from a pre-built model library, and the pre-trained target detection model is used to perform detection processing on the original image or the original image after image processing according to the target object to generate a detection result.

[0058] According to the order of each task command, the image processing results and / or detection results generated based on each task command are arranged in order to obtain the target detection reasoning result.

[0059] It should be noted that, on the page for receiving the task prompt, an example of the task prompt and the corresponding task command after parsing may be provided so that the user can input the task prompt according to the task prompt example and confirm whether the parsing is correct.

[0060] As a specific example, task prompt example 1: people detection inference using yolov9c. After parsing task prompt 1, the task command generated is: Pred1 = Det (image = 1.jpg, model = yolov9c_human).

[0061] As another specific example, task prompt example 2: Extract the trace image from the original image. The task command generated after parsing task prompt 2 is: Pred1 = Crop (target = target_img, raw = 1.jp g).

[0062] As a specific embodiment, when the user performs relevant target detection on the human body, hand, and smoke in the original image based on the above task prompt example, the task prompt input by the user may be “The three-stage yolov9c model was used to detect the human body, then the hand, and then the smoking”.

[0063] The GPT-3 model is used to perform semantic parsing on the task prompt "The three-stage yolov9c model was used to detect the human body, then the hand, and then the smoking", and the generated task flow command is:

[0064] Pred1=Det(image=1.jpg,model=yolov9c_human)

[0065] Pred2=Crop(target=Pred1,raw=1.jpg)

[0066] Pred3=Det(image=Pred2,model=yolov9c_hands)

[0067] Pred4=Crop(target=Pred3,raw=1.jpg)

[0068] Pred5=Det(image=Pred4, model=yolov9c_smoking).

[0069] The task commands are processed in sequence according to the above task flow commands. For task command 1: Pred1 = Det (image = 1.jpg, model = yolov9c_human). According to "Det", it can be determined that the task command type is a detection command. According to "image = 1.jpg", it can be determined that the target image is "1.jpg". According to "model = yolov9c_human", it can be determined that the target detection model is "yolov9c" and the target object is "human".

[0070] Therefore, the pre-trained yolov9c model can be called from the pre-built model library. Use the yolov9c model to detect the "human" in the image "1.jpg" and generate the corresponding detection results. Figure 2 shown.

[0071] For task command 2: Pred2 = Crop (target = Pred1, raw = 1.jpg). According to the target task type "Crop", it can be determined that the task command type is an image processing command, and the corresponding target image processing algorithm can also be determined to be an image segmentation algorithm. According to "target = Pred1", it can be determined that the result output by task command 1 is cropped. According to "raw = 1.jpg", it can be determined that the target image is "1.jpg".

[0072] Therefore, the image segmentation algorithm (target image processing algorithm) can be called from the pre-built image algorithm library. The image segmentation algorithm is used to segment "1.jpg" according to the output of task command 1 to generate the image processing result. Figure 2 shown.

[0073] For task command 3: Pred3 = Det (image = Pred2, model = yolov9c_hands). Task command 4: Pred4 = Crop (target = Pred3, raw = 1.jpg). Task command 5: Pred5 = Det (image = Pred4, model = yolov9c_smoking). I will not repeat them here. The results generated by task commands 3, 4 and 5 are shown in Figure 2 shown.

[0074] It should be noted that the task prompts of the present invention are described in English, and the task prompts may also be in Chinese, Korean, Japanese, etc. The language type used in the task prompts is not limited in the embodiments of the present invention.

[0075] In one embodiment of the present invention, Figure 3 As shown, generating a detection result may include:

[0076] S201, decoding the original result output by the pre-trained target detection model to obtain a decoding result;

[0077] S202, post-processing the decoding result, wherein the post-processing includes at least one of non-maximum suppression, inverse normalization, de-filling, and false detection and missed detection matching.

[0078] In order to improve the accuracy of the detection results, the embodiment of the present invention performs a post-processing operation on the original results output by the pre-trained target detection model.

[0079] Specifically, the original output of the pre-trained target detection model is decoded, and the obtained decoding result includes at least one of the detection frame, the category label corresponding to the detection frame, and the score corresponding to the detection frame. The decoding result is post-processed, including: filtering out unreliable detection frames according to a preset threshold and the score in the decoding result; performing non-maximum suppression (NMS) on the detection frames in the decoding result to eliminate duplicate detection frames, and converting the above-processed decoding result into a specific format to adapt to downstream tasks.

[0080] In one embodiment of the present invention, Figure 4 As shown, the target detection method may also include:

[0081] S301, receiving information fed back by the user based on the detection result;

[0082] S302: Adjust the target image processing algorithm and / or the pre-trained target detection model based on the feedback information.

[0083] In order to further improve the accuracy of target detection, the embodiment of the present invention also provides a user feedback interface.

[0084] Specifically, when the user finds a detection result with an error or inaccuracy in the labeling, the user labels the detection result. The user receives feedback based on the detection result, and adjusts the target image processing algorithm or the pre-trained target detection model based on the feedback information, or adjusts the target image processing algorithm and the pre-trained target detection model. Or when the accumulated feedback information reaches a preset value, the relevant model is optimized to adjust the parameters of the relevant model.

[0085] In one embodiment of the present invention, the target detection method may further include:

[0086] If the task command is a training command, then according to the training command, the target detection model to be trained is called from the pre-built model library, and the target detection model to be trained is trained using the original image or the original image after image processing to obtain a training result;

[0087] If the task command is a test command, the target detection model to be tested is called from a pre-built model library according to the test command, and the target detection model to be tested is tested using the original image or the original image after image processing to obtain a test result.

[0088] Specifically, if the target task type in the task command is a training type, the model architecture is selected, the training parameters are configured, and the training script is generated according to the task command. The specific image to be processed and the specific detection target can be determined according to the target image and the target object in the task command. Specifically, according to the training command, the target detection model to be trained is called from the pre-built model library, and the target detection model to be trained is trained using the original image or the original image after image processing to obtain the training result.

[0089] During model training, monitor the training loss, validation loss, accuracy, and recall in real time to ensure the effectiveness and stability of the training. Record the results of each epoch (each training dataset), model parameter changes, and any abnormalities. Logging is essential for subsequent analysis and debugging. Based on the monitoring results during training, adjust hyperparameters, use early stopping to avoid overfitting, and retrain the model to iterate and optimize the model.

[0090] Specifically, if the target task type in the task command is a test type, the model architecture is selected, the test parameters are configured, and the test script is generated according to the task command. The specific image to be processed and the specific detection target can be determined according to the target image and the target object in the task command. Specifically, according to the test command, the target detection model to be tested is called from the pre-built model library, and the target detection model to be tested is tested using the original image or the original image after image processing to obtain the test result.

[0091] In one embodiment of the present invention, Figure 5 and Figure 6 As shown, the BD_Net model includes a backbone network and auxiliary branches;

[0092] The backbone network includes the first input layer Input, the first convolution layer Conv, the second convolution layer Conv, the first feature extraction module DS-GELAN×4, the third convolution layer Conv, the first attention module CBAM, the deformable convolution layer Def_Conv, the second feature extraction module DS-GELAN×4, the fourth convolution layer Conv, the second attention module CBAM, the third feature extraction module DS-GELAN×4, the fifth convolution layer Conv, the third attention module CBAM, and the fourth feature extraction module DS-GELAN×4, which are connected in sequence; the second feature extraction module DS-GELAN×4 is connected to the first head head in sequence through the first attention mechanism SA, the first concatenation layer concat, the second attention mechanism SA, and the fifth feature extraction module DS-GELAN×4; the third feature extraction module DS-GELAN×4 is connected to the second head head in sequence through the third attention mechanism SA, the second concatenation layer concat, the fourth attention mechanism SA, and the sixth feature extraction module DS-GELAN×4. connection; the fourth feature extraction module DS-GELAN×4 is also connected to the third head head through the fifth attention mechanism SA, the pooling layer sppelan, the sixth attention mechanism SA, and the seventh feature extraction module DS-GELAN×4 in sequence; wherein, the pooling layer sppelan is connected to the second splicing layer concat through the first upsampling unsample, the second splicing layer concat is connected to the first splicing layer concat through the eighth feature extraction module DS-GELAN×4 and the second upsampling unsample, the fifth feature extraction module DS-GELAN×4 is connected to the sixth feature extraction module DS-GELAN×4 through the sixth convolutional layer Conv and the first feature fusion module FFM, the sixth feature extraction module DS-GELAN×4 is connected to the seventh feature extraction module DS-GELAN×4 through the seventh convolutional layer Conv and the second feature fusion module FFM, the third attention mechanism SA is connected to the first feature fusion module FFM, and the fifth attention mechanism SA is connected to the second feature fusion module FFM;

[0093] The auxiliary branch includes the second input layer Input, the seventh convolution layer Conv, the eighth convolution layer Conv, the ninth feature extraction module DS-GELAN×4, the attention and deformable convolution module CBAM+Def_Conv, the ninth convolution layer Conv, the first fusion layer CBFuse, the tenth feature extraction module DS-GELAN×4, the tenth convolution layer Conv, the second fusion layer Conv, the eleventh feature extraction module DS-GELAN×4, the eleventh convolution layer Conv, the third fusion layer CBFuse, the twelfth feature extraction module DS-GELAN×4, the twelfth convolution layer Conv, the third feature fusion module FFM, the thirteenth feature extraction module DS-GELAN×4, the thirteenth convolution layer Conv, the fourth feature fusion module FFM and the fourteenth feature extraction module DS-GELAN×4, which are connected in sequence. The third fusion layer CBFuse is connected to the fourth feature fusion module FFM, the twelfth feature extraction module DS-GELAN×4 is connected to the fourth head head, and the thirteenth feature extraction module DS-GELAN×4 is connected to the fourth head head. LAN×4 is connected to the fifth head head, the fourteenth feature extraction module DS-GELAN×4 is connected to the sixth head head, the auxiliary branch also includes the seventh attention mechanism SA, the eighth attention mechanism SA and the ninth attention mechanism SA, the seventh attention mechanism SA, the eighth attention mechanism SA and the ninth attention mechanism SA are also connected to the second fusion layer CBFuse, the seventh attention mechanism SA, the eighth attention mechanism SA and the ninth attention mechanism SA are also connected to the input end of the fourth feature fusion module FFM, the eighth attention mechanism SA is also connected to the first fusion layer CBFuse, the seventh attention mechanism SA is also connected to the third fusion layer CBFuse, and the seventh attention mechanism SA is also connected to the third feature fusion module FFM, wherein the second feature extraction module DS-GELAN×4 in the backbone network is connected to the seventh attention mechanism SA, the third feature extraction module DS-GELAN×4 in the backbone network is connected to the eighth attention mechanism SA, and the fourth feature extraction module DS-GELAN×4 in the backbone network is connected to the ninth attention mechanism SA.

[0094] It should be noted that the performance of the yolov9 series of algorithms is already better than other detection models, but due to the complexity of various defect data in the power distribution scenario, the extraction of comprehensive features is a challenging task for the model. At the same time, as the depth of the feature network increases, the network can extract more semantic information, but there is a risk of losing spatial detail information.

[0095] The related YOLOv9 model reduces the loss of feature information during the forward propagation of the model through programmable gradient information auxiliary branches. However, for the requirement of simultaneous detection of dozens of types of defects in substation scenarios, the extraction of comprehensive features is a challenging task for the YOLOv9 model. Especially when detecting small targets (such as human hands) and subtle changes (such as smoking), the related detection model faces the problem of insufficient accuracy and flexibility.

[0096] In order to solve the above problems, the embodiment of the present invention designs a BD_Net model. The overall architecture of the BD_Net model is as follows: Figure 5 and Figure 6 As shown, the BD_Net model in the embodiment of the present invention is improved based on the yolov9 model as the benchmark model. Based on the GELAN architecture of yolov9, a new feature extraction module DS-GELAN is introduced. The BD_Net model solves the problem of shallow information loss, insufficient feature extraction and fusion caused by network deepening.

[0097] Specifically, the backbone network of the BD_Net model is improved, and a new feature extraction module (DS-GELAN×4) is added. The DS-GELAN module is used to control model parameters and enhance feature extraction capabilities. The Convolutional Block Attention Module (CBAM) is improved, and the deformable convolution Def_Conv is introduced after the Convolutional Attention Module CBAM to further enhance the feature extraction capability. The newly added Feature Fusion Module (FFM) is combined with the Self-Attention mechanism (SA) to fuse features from different layers to improve the detection effect of downstream tasks. The auxiliary branch of the Programmable Gradient Information (PGI) of the BD_Net model is also improved, and the auxiliary branch of the programmable gradient information is enhanced to reduce information loss during forward propagation. In order to better fuse features and use gradient information to guide BD_Net model learning.

[0098] The main network and PGI auxiliary branch of the BD_Net model combine the deformable convolution Def_Conv and the convolutional attention module (CBAM). By adjusting the spatial attention module of CBAM, the feature extraction capability of the backbone network is improved. In addition, a new feature fusion module FFM is added to solve the problem of insufficient feature fusion by fusing feature maps of different levels and adding a self-attention mechanism SA to the jump connection of multi-level feature connections. The feature fusion module FFM is also integrated in the PGI auxiliary branch. The PGI auxiliary branch of yolov9 is modified to include a feature fusion network that performs unidirectional fusion of learned features, see Figure 6 .

[0099] Different from the traditional model, the embodiment of the present invention focuses on using a bidirectional feature fusion architecture to integrate richer semantic and spatial features, aiming to improve the accuracy of the backward gradient propagation parameters and improve the learning process of the backbone network.

[0100] It should be noted that the fusion layer (Channel-wise and Bilateral Fusion, CBFuse) can effectively integrate feature information from different sources or different levels through channel-level and bilateral fusion strategies.

[0101] In one embodiment of the present invention, Figure 7 As shown, the feature extraction module includes four DS_GELAN modules connected once, and the DS_GELAN module includes: a third input layer Input, a fourteenth convolution layer Conv, a fifteenth convolution layer Conv, a first cascade submodule b1, a second cascade submodule b2, a third cascade submodule b3 and a third concatenation layer concat, the output end of the third input layer Input is connected to the input ends of the fourteenth convolution layer Conv and the fifteenth convolution layer Conv respectively, and the output end of the fourteenth convolution layer Conv is connected to the input end of the third concatenation layer concat. The output end of the fifteenth convolutional layer Conv is connected to the input end of the first cascade submodule b1, the output end of the first cascade submodule b1 is connected to the input end of the third concatenation layer concat, the output end of the first cascade submodule b1 is connected to the input end of the second cascade submodule b2, the output end of the second cascade submodule b2 is connected to the input end of the third concatenation layer concat, the output end of the second cascade submodule b2 is connected to the input end of the third cascade submodule b3, and the output end of the third cascade submodule b3 is connected to the input end of the third concatenation layer concat.

[0102] Among them, the first cascade sub-module b1 includes a sixteenth convolutional layer Conv, a first depth-separable convolutional layer DSConv, a second depth-separable convolutional layer DSConv, a seventeenth convolutional layer Conv and a first addition layer add, the input ends of the sixteenth convolutional layer Conv and the first depth-separable convolutional layer DSConv are connected to the input end of the first cascade sub-module b1, the output end of the sixteenth convolutional layer Conv is connected to the input end of the first addition layer add, the output end of the first depth-separable convolutional layer DSConv is connected to the input end of the second depth-separable convolutional layer DSConv, the output end of the second depth-separable convolutional layer is connected to the input end of the seventeenth convolutional layer Conv, the output end of the seventeenth convolutional layer Conv is connected to the input end of the first addition layer add, and the output end of the first addition layer add is connected to the output end of the first cascade sub-module b1;

[0103] The second cascade sub-module b2 includes an eighteenth convolutional layer Conv, a nineteenth convolutional layer Conv, a third depthwise separable convolutional layer DSConv, a fourth depthwise separable convolutional layer DSConv, a twentieth convolutional layer Conv, and a second addition layer add, the input end of the eighteenth convolutional layer Conv is connected to the input end of the second cascade sub-module b2, the output end of the eighteenth convolutional layer Conv is connected to the input end of the nineteenth convolutional layer Conv, the output end of the nineteenth convolutional layer Conv is connected to the input end of the second addition layer add, the output end of the eighteenth convolutional layer Conv is connected to the input end of the third depthwise separable convolutional layer DSConv, the output end of the third depthwise separable convolutional layer DSConv is connected to the input end of the fourth depthwise separable convolutional layer DSConv, the output end of the fourth depthwise separable convolutional layer DSConv is connected to the input end of the twentieth convolutional layer Conv, the output end of the twentieth convolutional layer Conv is connected to the input end of the second addition layer add, and the output end of the second addition layer add is connected to the output end of the second cascade sub-module b2;

[0104] The third cascade sub-module b3 includes a twenty-first convolutional layer Conv, a twenty-second convolutional layer Conv, a fifth depthwise separable convolutional layer DSConv, a sixth depthwise separable convolutional layer DSConv, a twenty-third convolutional layer Conv, a seventh depthwise separable convolutional layer DSConv, an eighth depthwise separable convolutional layer DSConv and a twenty-fourth convolutional layer Conv, a fourth concatenation layer concat and a third addition layer add; the input ends of the twenty-first convolutional layer, the fifth depthwise separable convolutional layer DSConv and the seventh depthwise separable convolutional layer DSConv are connected to the input end of the third cascade sub-module b3, the output end of the twenty-first convolutional layer Conv is connected to the input end of the twenty-second convolutional layer Conv, the output end of the twenty-second convolutional layer Conv is connected to the input end of the third addition layer add, and the fifth depthwise separable convolutional layer DSConv is connected to the input end of the The output end of SConv is connected to the input end of the sixth depth-wise separable convolution layer DSConv, the output end of the sixth depth-wise separable convolution layer DSConv is connected to the input end of the twenty-third convolution layer Conv, the output end of the twenty-third convolution layer Conv is connected to the input end of the fourth concatenation layer concat, the output end of the seventh depth-wise separable convolution layer DSConv is connected to the input end of the eighth depth-wise separable convolution layer DSConv, the output end of the eighth depth-wise separable convolution layer DSConv is connected to the input end of the twenty-fourth convolution layer Conv, the output end of the twenty-fourth convolution layer Conv is connected to the input end of the fourth concatenation layer concat, the output end of the fourth concatenation layer concat is connected to the input end of the third addition layer add, and the output end of the third addition layer add is connected to the output end of the third cascade sub-module b3.

[0105] The DS_GELAN module of the embodiment of the present invention is composed of multiple modules that extract features in sequence, and the features of each layer are added in parallel. The three smaller modules in the DS_GELAN module use depthwise separable convolution DSConv to control the model complexity while maintaining the feature extraction capability. In addition, the width of the DS_GELAN module is also increased. The DS_GELAN module not only enhances the depth of the backbone network and extracts more semantic information, but also broadens the width of the model and captures more spatial detail features. GELAN takes into account the feature extraction capability and the network depth, width and model complexity.

[0106] The DS-GELAN module has two main branches. The left branch uses 3*3 convolution Conv, BN and SiLU (BN and SiLU are not shown) to enhance feature extraction, increase the number of channels without downsampling, and then restore to the original dimension through a 1*1 convolution layer Conv, ensuring the extraction of effective features while retaining the control of the feature map dimension for subsequent operations. In the right branch, the 3*3 convolution layer Conv is also used for feature extraction, and then the outputs of the three cascade submodules (b1, b2, b3) are added and merged to generate the heaviest feature map of the right branch. Finally, the feature maps of the two branches are spliced ​​in the channel dimension to generate the output of the DS-GELAN module. The cascade submodule contains three branches, where the first cascade submodule b1 and the second cascade submodule b2 branches both produce two outputs, which are used for feature addition and input of subsequent modules respectively, and the third cascade submodule b3 generates a single output.

[0107] The initial convolution Conv in the right branch of the DS-GELAN module resizes the feature map to H×W×2C, and the 1×1 convolution Conv within the first cascade submodule b1 changes the feature map size to H×W×C. The right branch of b1 is characterized by a structure that combines two 3×3 depthwise separable convolutions DSConv and one 1×1 convolution Conv. The separable convolution DSConv helps with feature extraction, while the 1×1 convolution Conv reduces the number of channels to achieve a feature map size of H×W×C, which is beneficial for subsequent addition operations. Using 1×1 convolution Conv for channel compression ensures the consistency of the feature map size of the internal module and simplifies subsequent operations.

[0108] The first cascade submodule b1 and the second cascade submodule b2 are used for conventional feature extraction. The second cascade submodule b2 has a similar feature extraction structure to the first cascade submodule b1 after the convolution compression channel, and improves the network by superimposing two similar structures to extract features. b3 uses three branches to enhance the network structure and improve the comprehensiveness of the extracted information, thereby avoiding the insufficient feature extraction of the first cascade submodule b1 and the second cascade submodule b2. The input feature map F of the third cascade submodule b3 is derived from the output of b2, and F is feature extracted by 3×3 convolution Conv, and then channel control is performed using 1×1 convolution Conv. The remaining two branches use depthwise separable convolution DSConv and 1×1 convolution Conv for feature extraction respectively. Subsequently, the features extracted from the two branches are connected in the channel dimension and added to the output of the left branch, and the resulting output feature map has a size of H×W×C. Among them, the middle branch uses 3×3 depth-wise separable convolution DSConv, while the right branch uses 5×5 depth-wise separable convolution DSConv to expand the receptive field of the network, integrate features of different receptive fields, and reduce the texture feature loss of smaller targets.

[0109] In one embodiment of the present invention, Figure 8 As shown, the feature fusion module FFM includes a twenty-fifth convolutional layer Conv, a fifth concatenation layer concat, a twenty-sixth convolutional layer Conv, a twenty-seventh convolutional layer Conv, a sixth concatenation layer concat, a twenty-eighth convolutional layer Conv and an output layer output which are connected in sequence. The twenty-fifth convolutional layer Conv is connected to the first input terminal and the second input terminal of the feature fusion module FFM. The feature fusion module also includes a twenty-ninth convolutional layer Conv. The input terminal of the twenty-ninth convolutional layer Conv is connected to the third input terminal of the feature fusion module. The output terminal of the twenty-ninth convolutional layer Conv is connected to the input terminal of the sixth concatenation layer concat, and the output layer output is connected to the output terminal of the feature fusion module.

[0110] In an embodiment of the present invention, the feature fusion module FFM combines feature maps from three dimensions, the feature map from the previous layer of the feature fusion module FFM, the same-level feature maps sampled from top to bottom in the feature fusion network (feature fusion network), and the same-level feature maps in the backbone network.

[0111] The feature map of the previous layer of the feature fusion module FFM and the feature map sampled from top to bottom are concat in the channel dimension, and then a 3×3 convolution Conv operation is performed to extract features from the fused feature map. Then a 1×1 convolution Conv is used to compress the channel dimension and fuse it with the backbone network feature map processed by 3×3 convolution to enrich the feature map with multi-dimensional features. Finally, the number of channels is reduced through a 1×1 convolution to obtain the output features.

[0112] The embodiment of the present invention provides visual variable programming defect detection training, training and detection reasoning methods, and the visual variable programming electric scene defect detection training and reasoning system uses the context learning ability of a large language model to generate Python modular programs, and then executes these programs to achieve flexible defect detection and efficient solutions. Among them, the algorithm involved is the BD_Net model defect detection algorithm based on the yolov9 framework, which can efficiently and accurately detect the location of smoke, bird nests, foreign objects, broken insulators, and broken wires in the substation scene, give different levels of alarm signals, and overcome the limitations of existing detection algorithms.

[0113] The invention provides a target detection system.

[0114] Fig. 9 FIG. 1 is a schematic diagram of a target detection system according to an embodiment of the present invention. Fig. 9 As shown, the target detection system 100 includes a receiving module 10, a semantic parsing module 20, a processing module 30 and an output module 40.

[0115] Among them, the receiving module 10 is used to receive the original image and the task prompt; the semantic parsing module 20 is used to perform semantic analysis on the task prompt and generate a task flow command, wherein the task flow command includes at least one task command; the processing module 30 is used to call the target image processing algorithm from a pre-built image algorithm library according to the image processing command when the task command is an image processing command, and use the target image processing algorithm to perform image processing on the original image to generate an image processing result; when the task command is a detection command, call a pre-trained target detection model from a pre-built model library according to the detection command, and use the pre-trained target detection model to perform detection processing on the original image or the original image after image processing according to the target object to generate a detection result; the output module 40 is used to obtain the target detection inference result according to the image processing result and / or detection result generated based on each task command in the task flow command.

[0116] It should be noted that when dealing with such complex tasks, a large amount of code is often written for image preprocessing, training, testing, and reasoning tasks based on the characteristics of the defect targets. Different defect targets have different detection modes, and the reasoning and training frameworks are also different. It is necessary to write different codes multiple times to build the framework, which cannot be automated. This makes it challenging to expand to the complex defect detection tasks that these systems are ultimately expected to perform.

[0117] like Fig.10 As shown, the semantic parsing module in the target detection system 100 is used to call the relevant large language model to perform semantic analysis on the task prompt and generate the task flow command. The processing module 30 includes an image processing unit, a model unit and a post-processing unit. The image processing unit is provided with a variety of algorithm models such as image enhancement algorithm, image normalization algorithm, image denoising algorithm, image blurring algorithm, etc. The model unit includes yolov8 model, yolov9 model, BD_Net model, gdino model, etc. The post-processing unit includes non-maximum suppression algorithm, anti-normalization algorithm, de-filling algorithm, false detection and missed detection matching algorithm, etc.

[0118] The embodiment of the present invention preliminarily builds and maintains an image algorithm library including various image processing, a model library including various models, and an algorithm library including various post-processing in the image processing unit, the model unit, and the post-processing unit, respectively, so that the image processing unit, the model unit, and the post-processing unit can load and call the relevant image algorithm, model, and post-processing algorithm, respectively.

[0119] In order to solve the lack of interpretability and adaptability of various defect detections in the target detection process, an embodiment of the present invention receives an original image and a task prompt, and detects the original image according to the task prompt. Specifically, a large language model is used to perform semantic analysis on the task prompt, and a task flow command including at least one task command is generated. According to the task command in the task flow command, the relevant image algorithms, models and post-processing algorithms in the image processing unit, model unit and post-processing unit are sequentially loaded or called to process the original image, and the task processing results corresponding to each task command are generated, so that each processing process of the target detection is visualized, and the interpretability and adaptability of the target detection process are achieved.

[0120] It should be noted that other specific implementations of the target detection system provided in the embodiment of the present invention may refer to other specific implementations of the target detection method in the above embodiment of the present invention.

[0121] The target detection system of the embodiment of the present invention processes the original image in combination with the task prompts input by the user, so that the user can interact with the reasoning process, making the reasoning process explainable and adaptable while also improving the efficiency of model training, testing and detection.

[0122] The invention provides a controller.

[0123] In this embodiment, the controller may include a memory and a processor. The memory stores a computer program. When the computer program is executed by the processor, the target detection method as described above is implemented.

[0124] Fig.11 4 is a structural block diagram of a controller according to an embodiment of the present invention.

[0125] like Fig.11 As shown, the controller 500 includes: a processor 501 and a memory 503. The processor 501 and the memory 503 are connected, such as through a bus 502. Optionally, the controller 500 may also include a transceiver 504. It should be noted that in actual applications, the transceiver 504 is not limited to one, and the structure of the controller 500 does not constitute a limitation on the embodiments of the present invention.

[0126] Processor 501 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of the present invention. Processor 501 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.

[0127] The bus 502 may include a path to transmit information between the above components. The bus 502 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 502 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.11 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0128] The memory 503 is used to store a computer program corresponding to the target detection method of the above embodiment of the present invention, and the computer program is controlled and executed by the processor 501. The processor 501 is used to execute the computer program stored in the memory 503 to implement the content shown in the above method embodiment.

[0129] Among them, the controller 500 includes but is not limited to: mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.11 The controller 500 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0130] The controller of an embodiment of the present invention processes the original image based on the above-mentioned target detection method, so that the user can interact with the reasoning process, making the reasoning process explainable and adaptable while also improving the efficiency of model training, testing and detection.

[0131] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection portion with one or more wirings (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0132] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0133] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0134] In the description of the present invention, it is to be understood that the terms “center”, “longitudinal”, “lateral”, “length”, “width”, “thickness”, “up”, “down”, “front”, “back”, “left”, “right”, “vertical”, “horizontal”, “top”, “bottom”, “inside”, “outside”, “clockwise”, “counterclockwise”, “axial”, “radial”, “circumferential”, etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the present invention.

[0135] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0136] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements, unless otherwise clearly defined. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0137] In the present invention, unless otherwise clearly specified and limited, a first feature being "above" or "below" a second feature may mean that the first and second features are in direct contact, or the first and second features are in indirect contact through an intermediate medium. Moreover, a first feature being "above", "above" or "above" a second feature may mean that the first feature is directly above or obliquely above the second feature, or simply means that the first feature is higher in level than the second feature. A first feature being "below", "below" or "below" a second feature may mean that the first feature is directly below or obliquely below the second feature, or simply means that the first feature is lower in level than the second feature.

[0138] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A target detection method, characterized in that: The method comprises: Receive original images and task prompts; Performing semantic analysis on the task prompt to generate a task flow command, wherein the task flow command includes at least one task command; If the task command is an image processing command, then according to the image processing command, a target image processing algorithm is called from a pre-built image algorithm library, and the original image is processed by using the target image processing algorithm to generate an image processing result; If the task command is a detection command, then according to the detection command, a pre-trained target detection model is called from a pre-built model library, and the pre-trained target detection model is used to perform detection processing on the original image or the original image after image processing according to the target object to generate a detection result; According to the image processing results and / or the detection results generated based on each of the task commands in the task flow command, a target detection inference result is obtained.

2. The target detection method according to claim 1, characterized in that: The method further comprises: If the task command is a training command, then according to the training command, the target detection model to be trained is called from a pre-built model library, and the target detection model to be trained is trained using the original image or the original image after image processing to obtain a training result; If the task command is a test command, then according to the test command, the target detection model to be tested is called from a pre-built model library, and the target detection model to be tested is tested using the original image or the original image after image processing to obtain a test result.

3. The target detection method according to claim 1, characterized in that: Generating the detection result comprises: Decoding the original result output by the pre-trained target detection model to obtain a decoding result; The decoding result is post-processed, wherein the post-processing includes at least one of non-maximum suppression, inverse normalization, de-filling, and false detection and missed detection matching.

4. The target detection method according to claim 1, characterized in that: The method further comprises: Receiving information fed back by a user based on the detection result; Based on the feedback information, the target image processing algorithm and / or the pre-trained target detection model are adjusted.

5. The target detection method according to claim 1 or 2, characterized in that: The task command includes at least one of a target task type, a target image processing algorithm, a target detection model, a target image, and a target object, wherein the target task type includes at least one of an image processing type, a detection type, a training type, and a test type; The pre-built model library includes at least one of the yolov8 model, the yolov9 model, the BD_Net model, and the gdino model; The pre-built image algorithm library includes at least one of an image enhancement algorithm, an image edge detection algorithm, an image segmentation algorithm, an image augmentation algorithm, and the like.

6. The target detection method according to claim 5, characterized in that: The BD_Net model includes a backbone network and auxiliary branches; The backbone network includes a first input layer, a first convolution layer, a second convolution layer, a first feature extraction module, a third convolution layer, a first attention module, a deformable convolution layer, a second feature extraction module, a fourth convolution layer, a second attention module, a third feature extraction module, a fifth convolution layer, a third attention module, and a fourth feature extraction module, which are connected in sequence; the second feature extraction module is connected to the first head in sequence through a first attention mechanism, a first splicing layer, a second attention mechanism, and a fifth feature extraction module; the third feature extraction module is connected to the second head in sequence through a third attention mechanism, a second splicing layer, a fourth attention mechanism, and a sixth feature extraction module; the fourth feature extraction module is connected to the second head in sequence through a third attention mechanism, a second splicing layer, a fourth attention mechanism, and a sixth feature extraction module; The extraction module is also connected to the third head through the fifth attention mechanism, the pooling layer, the sixth attention mechanism, and the seventh feature extraction module in sequence; wherein the pooling layer is connected to the second splicing layer through the first upsampling, the second splicing layer is connected to the first splicing layer through the eighth feature extraction module and the second upsampling, the fifth feature extraction module is connected to the sixth feature extraction module through the sixth convolution layer and the first feature fusion module, the sixth feature extraction module is connected to the seventh feature extraction module through the seventh convolution layer and the second feature fusion module, the third attention mechanism is connected to the first feature fusion module, and the fifth attention mechanism is connected to the second feature fusion module; The auxiliary branch includes a second input layer, a seventh convolution layer, an eighth convolution layer, a ninth feature extraction module, an attention and deformable convolution module, a ninth convolution layer, a first fusion layer, a tenth feature extraction module, a tenth convolution layer, a second fusion layer, an eleventh feature extraction module, an eleventh convolution layer, a third fusion layer, a twelfth feature extraction module, a twelfth convolution layer, a third feature fusion module, a thirteenth feature extraction module, a thirteenth convolution layer, a fourth feature fusion module and a fourteenth feature extraction module, the third fusion layer is connected to the fourth feature fusion module, the twelfth feature extraction module is connected to the fourth head, the thirteenth feature extraction module is connected to the fifth head, and the fourteenth feature extraction module is connected to the sixth head. The auxiliary branch also includes a seventh attention mechanism, a first fusion layer, a second fusion layer, a tenth feature extraction module, a tenth convolution layer, a second fusion layer, a twelfth feature extraction module, a twelfth convolution layer, a third feature fusion module, a thirteenth feature extraction module, a thirteenth convolution layer, a fourth feature fusion module and a fourteenth feature extraction module. The third fusion layer is connected to the fourth feature fusion module, the twelfth feature extraction module is connected to the fourth head, the thirteenth feature extraction module is connected to the fifth head, and the fourteenth feature extraction module is connected to the sixth head. The eighth attention mechanism and the ninth attention mechanism, the seventh attention mechanism, the eighth attention mechanism and the ninth attention mechanism are also connected to the second fusion layer, the seventh attention mechanism, the eighth attention mechanism and the ninth attention mechanism are also connected to the input end of the fourth feature fusion module, the eighth attention mechanism is also connected to the first fusion layer, the seventh attention mechanism is also connected to the third fusion layer, the seventh attention mechanism is also connected to the third feature fusion module, wherein the second feature extraction module in the backbone network is connected to the seventh attention mechanism, the third feature extraction module in the backbone network is connected to the eighth attention mechanism, and the fourth feature extraction module in the backbone network is connected to the ninth attention mechanism.

7. The target detection method according to claim 6, characterized in that: The feature extraction module includes 4 DS_GELAN modules connected once, and the DS_GELAN module includes: a third input layer, a fourteenth convolutional layer, a fifteenth convolutional layer, a first cascade submodule, a second cascade submodule, a third cascade submodule and a third splicing layer, the output end of the third input layer is respectively connected to the input ends of the fourteenth convolutional layer and the fifteenth convolutional layer, the output end of the fourteenth convolutional layer is connected to the input end of the third splicing layer, the output end of the fifteenth convolutional layer is connected to the input end of the first cascade submodule, the output end of the first cascade submodule is connected to the input end of the third splicing layer, the output end of the first cascade submodule is connected to the input end of the second cascade submodule, the output end of the second cascade submodule is connected to the input end of the third splicing layer, the output end of the second cascade submodule is connected to the input end of the third cascade submodule, and the output end of the third cascade submodule is connected to the input end of the third splicing layer. Wherein, the first cascade submodule includes a sixteenth convolutional layer, a first depthwise separable convolutional layer, a second depthwise separable convolutional layer, a seventeenth convolutional layer and a first addition layer, the input ends of the sixteenth convolutional layer and the first depthwise separable convolutional layer are connected to the input end of the first cascade submodule, the output end of the sixteenth convolutional layer is connected to the input end of the first addition layer, the output end of the first depthwise separable convolutional layer is connected to the input end of the second depthwise separable convolutional layer, the output end of the second depthwise separable convolutional layer is connected to the input end of the seventeenth convolutional layer, the output end of the seventeenth convolutional layer is connected to the input end of the first addition layer, and the output end of the first addition layer is connected to the output end of the first cascade submodule; The second cascade submodule includes an eighteenth convolutional layer, a nineteenth convolutional layer, a third depthwise separable convolutional layer, a fourth depthwise separable convolutional layer, a twentieth convolutional layer and a second addition layer, the input end of the eighteenth convolutional layer is connected to the input end of the second cascade submodule, the output end of the eighteenth convolutional layer is connected to the input end of the nineteenth convolutional layer, the output end of the nineteenth convolutional layer is connected to the input end of the second addition layer, the output end of the eighteenth convolutional layer is connected to the input end of the third depthwise separable convolutional layer, the output end of the third depthwise separable convolutional layer is connected to the input end of the fourth depthwise separable convolutional layer, the output end of the fourth depthwise separable convolutional layer is connected to the input end of the twentieth convolutional layer, the output end of the twentieth convolutional layer is connected to the input end of the second addition layer, and the output end of the second addition layer is connected to the output end of the second cascade submodule; The third cascade submodule includes a twenty-first convolutional layer, a twenty-second convolutional layer, a fifth depthwise separable convolutional layer, a sixth depthwise separable convolutional layer, a twenty-third convolutional layer, a seventh depthwise separable convolutional layer, an eighth depthwise separable convolutional layer and a twenty-fourth convolutional layer, a fourth splicing layer and a third addition layer, the input ends of the twenty-first convolutional layer, the fifth depthwise separable convolutional layer and the seventh depthwise separable convolutional layer are connected to the input end of the third cascade submodule, the output end of the twenty-first convolutional layer is connected to the input end of the twenty-second convolutional layer, the output end of the twenty-second convolutional layer is connected to the input end of the third addition layer, and the output end of the fifth depthwise separable convolutional layer is connected to the input end of the The input end of the sixth depthwise separable convolutional layer is connected, the output end of the sixth depthwise separable convolutional layer is connected to the input end of the twenty-third convolutional layer, the output end of the twenty-third convolutional layer is connected to the input end of the fourth splicing layer, the output end of the seventh depthwise separable convolutional layer is connected to the input end of the eighth depthwise separable convolutional layer, the output end of the eighth depthwise separable convolutional layer is connected to the input end of the twenty-fourth convolutional layer, the output end of the twenty-fourth convolutional layer is connected to the input end of the fourth splicing layer, the output end of the fourth splicing layer is connected to the input end of the third addition layer, and the output end of the third addition layer is connected to the output end of the third cascade sub-module.

8. The target detection method according to claim 6, characterized in that: The feature fusion module includes a twenty-fifth convolutional layer, a fifth splicing layer, a twenty-sixth convolutional layer, a twenty-seventh convolutional layer, a sixth splicing layer, a twenty-eighth convolutional layer and an output layer connected in sequence, the twenty-fifth convolutional layer is connected to the first input end and the second input end of the feature fusion module, the feature fusion module also includes a twenty-ninth convolutional layer, the input end of the twenty-ninth convolutional layer is connected to the third input end of the feature fusion module, the output end of the twenty-ninth convolutional layer is connected to the input end of the sixth splicing layer, and the output layer is connected to the output end of the feature fusion module.

9. A target detection system, characterized in that: The system comprises: A receiving module, used for receiving original images and task prompts; A semantic analysis module, used for performing semantic analysis on the task prompt to generate a task flow command, wherein the task flow command includes at least one task command; a processing module, for, when the task command is an image processing command, calling a target image processing algorithm from a pre-built image algorithm library according to the image processing command, performing image processing on the original image using the target image processing algorithm, and generating an image processing result; and, when the task command is a detection command, calling a pre-trained target detection model from a pre-built model library according to the detection command, performing detection processing on the original image or the original image after image processing according to the target object using the pre-trained target detection model, and generating a detection result; The output module is used to obtain the target detection reasoning result according to the image processing result and / or the detection result generated based on each task command in the task flow command.

10. A controller, comprising a memory and a processor, wherein a computer program is stored in the memory, characterized in that: When the computer program is executed by the processor, the target detection method according to any one of claims 1 to 8 is implemented.

Images