Fish polyculture judgment method based on lightweight multi-task image recognition network and knowledge graph

Abstract

The invention relates to the technical field of intelligent cultivation, and provides a fish polyculture judgment method based on a lightweight multi-task image recognition network and a knowledge graph, and the method comprises the steps: obtaining image data and sensor data, and carrying out the preprocessing of the image data, and obtaining a to-be-recognized image; inputting the to-be-recognized image into a pre-trained image recognition model to generate visual perception data; performing weighted fusion on the visual perception data and the sensor data to form a comprehensive feature vector; and inputting the comprehensive feature vector into a pre-trained knowledge graph matching model, calculating a matching degree between the comprehensive feature vector and a polyculture rule in a knowledge graph according to a dynamic rule matching mechanism, and determining whether polyculture can be performed according to the matching degree. The problem that the survival rate of existing fish polyculture is low is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent aquaculture, and particularly to a method for judging fish polyculture based on a lightweight multi-task image recognition network and a knowledge graph. Background Art

[0002] In the process of home fish farming, how to reasonably mix different species of fish for polyculture has always been an urgent problem to be solved. The traditional method of home fish polyculture mainly relies on artificial experience, and it is impossible to accurately judge whether different species of fish are suitable for polyculture, resulting in certain blindness and risks.

[0003] There are significant differences in body size, growth cycle, swimming level, temperament, suitable water temperature and pH value, etc. among different species of fish. If the fish polyculture is improper, problems such as big fish eating small fish or fry, and unsuitable living environment for fish may occur, affecting the growth and survival of fish. In addition, different fish have different requirements for environmental conditions. If the environment does not match, it will also affect the survival state of fish. Therefore, there is an urgent need for a new method of fish polyculture that can accurately judge whether different species of fish are suitable for polyculture according to information such as the species, body size, swimming level, temperament, and environmental requirements for survival of fish, so as to improve the efficiency and survival rate of fish farming. Summary of the Invention

[0004] The present invention provides a method for judging fish polyculture based on a lightweight multi-task image recognition network and a knowledge graph to solve the problem of low survival rate of existing fish polyculture.

[0005] The present invention provides a method for judging fish polyculture based on a lightweight multi-task image recognition network and a knowledge graph, including: Obtain image data and sensor data, and preprocess the image data to obtain an image to be recognized; Input the image to be recognized into a pre-trained image recognition model to generate visual perception data; Perform weighted fusion on the visual perception data and the sensor data to form a comprehensive feature vector; Input the comprehensive feature vector into a pre-trained knowledge graph matching model, and calculate the matching degree between the comprehensive feature vector and the polyculture rules in the knowledge graph according to the dynamic rule matching mechanism, and determine whether polyculture is possible according to the matching degree; Wherein, the pre-trained image recognition model is obtained by training a preset deep learning object detection model with fish images with annotation information, and the pre-trained knowledge graph matching model is obtained by training a preset deep feature matching model with a data set having a corresponding relationship between the comprehensive feature vector and the knowledge graph.

[0006] A method for judging polyculture of fish based on a lightweight multi-task image recognition network and a knowledge graph provided by the present invention. The preprocessing of the image data to obtain the image to be recognized specifically includes: Obtain images of polyculture of multiple fish in different environments through an image acquisition device; Perform data cleaning, data filtering, and data repair on the images of polyculture of multiple fish in different environments to obtain primary image data; Convert the format of the primary image data, denoise the image, normalize the image, and enhance the image to obtain the image to be recognized.

[0007] A method for judging polyculture of fish based on a lightweight multi-task image recognition network and a knowledge graph provided by the present invention. The pre-trained image recognition model is obtained by training a preset deep learning object detection model with fish images with annotation information, specifically including: Obtain fish images annotated with fish species, body size, and swimming level information, and divide them into a training set, a validation set, and a test set according to a set ratio; Train a preset deep learning object detection model with the training set, and adjust the hyperparameters with the validation set to obtain an image recognition model; Test and evaluate the performance of the image recognition model with the test set and perform fine-tuning to obtain the final image recognition model; Among them, the image recognition model is constructed based on a lightweight convolutional neural network and a multi-task learning framework, sharing the underlying feature extraction network within the framework, and setting specific output layers for each task at the top layer. Through feature reuse and linear transformation, lightweight multi-task parallel processing is completed.

[0008] A method for judging polyculture of fish based on a lightweight multi-task image recognition network and a knowledge graph provided by the present invention. The inputting of the image to be recognized into the pre-trained image recognition model to generate visual perception data specifically includes: Input the image to be recognized into the pre-trained image recognition model; Perform feature fusion and target localization through the image recognition model to determine the target species, body size, and swimming level, and obtain visual perception data after optimization by the total loss function.

[0009] A method for judging polyculture of fish based on a lightweight multi-task image recognition network and a knowledge graph provided by the present invention. The pre-trained knowledge graph matching model is obtained by training a preset deep feature matching model with a dataset with the corresponding relationship between the comprehensive feature vector and the knowledge graph, specifically including: Obtain fish breeding data and perform data cleaning and semantic alignment to obtain structured data and unstructured data; Based on the structured data and unstructured data, a core entity and relationship system is predefined through ontology, and entity relationship joint extraction is realized by combining rule-driven and deep learning to construct a knowledge graph; Cross-modal associate the comprehensive feature vectors of the knowledge graph to train a preset deep feature matching model to obtain a knowledge graph matching model.

[0010] According to a method for judging fish polyculture based on a lightweight multi-task image recognition network and a knowledge graph provided by the present invention, inputting the comprehensive feature vectors into a pre-trained knowledge graph matching model, and calculating the matching degree between the comprehensive feature vectors and the polyculture rules in the knowledge graph according to a dynamic rule matching mechanism, and determining whether polyculture is possible according to the matching degree, specifically including: Input the comprehensive feature vectors into the knowledge graph matching model; Match the comprehensive feature vectors with the polyculture knowledge graph of farmed fish in the knowledge graph matching model; according to various feature values in the comprehensive feature vectors, locate the corresponding fish species nodes, feature nodes and associated polyculture rule nodes in the knowledge graph; According to the dynamic rule matching mechanism, calculate the matching degree between the comprehensive feature vectors and the polyculture rules in the knowledge graph, and judge whether the farmed fish can be polycultured according to the matching degree result.

[0011] The present invention also provides a system for judging fish polyculture based on a lightweight multi-task image recognition network and a knowledge graph, including: A data acquisition module, configured to acquire image data and sensor data, and preprocess the image data to obtain an image to be recognized; A visual perception data generation module, configured to input the image to be recognized into a pre-trained image recognition model to generate visual perception data; A fusion module, configured to perform weighted fusion on the visual perception data and the sensor data to form comprehensive feature vectors; A matching module, configured to input the comprehensive feature vectors into a pre-trained knowledge graph matching model, calculate the matching degree between the comprehensive feature vectors and the polyculture rules in the knowledge graph according to a dynamic rule matching mechanism, and determine whether polyculture is possible according to the matching degree; Wherein, the pre-trained image recognition model is obtained by training a preset deep learning object detection model with fish images with annotation information, and the pre-trained knowledge graph matching model is obtained by training a preset deep feature matching model with a data set with the corresponding relationship between comprehensive feature vectors and the knowledge graph.

[0012] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the method for judging fish polyculture based on a lightweight multi-task image recognition network and a knowledge graph as described in any one of the above is implemented.

[0013] The present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method for judging fish polyculture based on a lightweight multi-task image recognition network and a knowledge graph as described in any one of the above is implemented.

[0014] The present invention also provides a computer program product, including a computer program. When the computer program is executed by a processor, the method for judging fish polyculture based on a lightweight multi-task image recognition network and a knowledge graph as described in any one of the above is implemented.

[0015] The method for judging fish polyculture based on a lightweight multi-task image recognition network and a knowledge graph provided by the present invention obtains information such as the species, body size, and swimming level of fish by performing image recognition on the fish, and combines information such as the temperament, water temperature, and pH value of different fish stored in the fish farming knowledge graph to determine whether different species of fish are suitable for polyculture through multi-modal data fusion; it can accurately judge whether different species of fish are suitable for polyculture, avoid problems such as big fish eating small fish or fry, and unsuitable living environments for fish, and improve the efficiency and survival rate of fish farming. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0017] Figure 1 It is a schematic flowchart of the method for judging fish polyculture based on a lightweight multi-task image recognition network and a knowledge graph provided by the present invention.

[0018] Figure 2 It is an overall architecture diagram of the method for judging fish polyculture based on a lightweight multi-task image recognition network and a knowledge graph provided by the present invention.

[0019] Figure 3 It is a schematic diagram of the backbone network of the image recognition model provided by the present invention.

[0020] Figure 4 It is a schematic diagram of feature fusion provided by the present invention.

[0021] Figure 5 It is a schematic diagram of the multi-task network architecture provided by the present invention.

[0022] Figure 6 It is a schematic diagram of the module connection of the fish polyculture judgment system based on the lightweight multi-task image recognition network and knowledge graph provided by the present invention.

[0023] Figure 7 It is a schematic diagram of the structure of the electronic device provided by the present invention.

[0024] Reference numerals: 110: data acquisition module; 120: visual perception data generation module; 130: fusion module; 140: matching module; 710: processor; 720: communication interface; 730: memory; 740: communication bus. Detailed implementation manners

[0025] In order to make the objectives, technical solutions and advantages of the present invention clearer, the technical solutions in the present invention will be clearly and completely described below with reference to the accompanying drawings in the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present invention without making creative efforts shall fall within the protection scope of the present invention.

[0026] When culturing existing household ornamental fish, different fish have different requirements for environmental conditions. If the environment does not match, it will also affect the survival status of the fish. Therefore, it is necessary to accurately judge whether different types of fish are suitable for mixed culture according to information such as the species, body size, swimming level, temperament, and survival environment requirements of the fish, so as to improve the efficiency and survival rate of fish culture. In view of the problems of blindness, poor environmental adaptability, and low survival rate caused by traditional artificial experience in mixed culture of household fish, the present invention first proposes to optimize the structure of the YOLOv10 target detection algorithm: by using the GhostBottleneck module and designing an attention mechanism and loss function for the task of judging fish mixed culture, the recognition accuracy of fish targets in a complex water environment is improved, and a multi-task processing framework is constructed to detect the species, dynamic body size parameters (body length / body width), and swimming levels (upper / middle / lower layers) of fish in real time; at the same time, a structured knowledge graph of household fish is constructed, covering multi-dimensional data such as the temperament aggression of fish, suitable water temperature / pH range, growth cycle, and symbiotic compatibility rules. The image recognition results are dynamically matched with the knowledge graph rules, and the compatibility of mixed culture is judged through a conflict detection algorithm (such as weighted calculation based on similarity threshold and conflict factor), and real-time warnings are given for incompatible combinations with predation risks, environmental parameter conflicts, or space competition. The innovation of the present invention lies in dynamically combining visual perception (image recognition) with ecological rules (knowledge graph) for the first time, breaking through the limitations of a single data source. By improving the real-time detection accuracy of YOLOv10 and the quantization rule engine of the knowledge graph, the accuracy of judging the mixed culture of household fish is improved.

[0027] The following combines Figure 1 and Figure 2 to describe a method for judging fish mixed culture based on a lightweight multi-task image recognition network and a knowledge graph according to the present invention, including: Step 100, obtain image data and sensor data, and preprocess the image data to obtain an image to be recognized.

[0028] In the present invention, it is necessary to select a suitable image acquisition device, such as a high-resolution camera or a mobile phone camera with a professional shooting mode, to ensure the clarity and richness of details of the captured image. During the shooting process, the influence of different lighting conditions on the image quality needs to be considered. Therefore, it is necessary to shoot in various lighting environments such as strong light, weak light, and natural light to enhance the adaptability of the model to different light conditions. At the same time, in order to comprehensively capture the morphological characteristics of the fish, it is necessary to shoot from multiple angles, including different perspectives such as the front, side, and inclined planes, and shoot at different distances to obtain all-round information of the fish. During the image acquisition process, it is also necessary to pay attention to maintaining the stability of shooting and the accuracy of focus to avoid image quality degradation caused by blurring or out-of-focus.

[0029] Images of farmed fish: Take images of different farmed fish in different environments such as fish tanks or aquariums; Images of swimming levels: Take images of mixed cultures of multiple fish species.

[0030] After obtaining the initial image data, it is necessary to preprocess the image data. In the present invention, data deduplication is performed by comparing the hash values or feature vectors of the images to identify and delete duplicate images, avoiding the introduction of redundant information in subsequent processing and analysis. In addition, deduplication can also be performed based on file names, image metadata (such as shooting time, location, device, etc.) to further improve the purity of the data; Data filtering is to screen the images to eliminate those with poor quality such as blurred, severely blocked, or too low resolution, as well as images that do not meet the requirements, such as images with non-standard sizes or incomplete content, to ensure the overall quality of the dataset; Data repair is to use image repair algorithms to repair defects in the images, such as scratches, spots, noise, etc., to improve the clarity and integrity of the images, making them more conducive to subsequent feature extraction and analysis. After the image cleaning step, the obtained image data is more standardized, complete, and representative, laying a foundation for subsequent image data preprocessing and deep learning model training.

[0031] After completing the image cleaning, it is necessary to preprocess the image data next to make it meet the requirements of the subsequent deep learning model input and improve the performance and robustness of the model. The image data preprocessing mainly includes the following aspects: Image format conversion, converting the collected images into the format required by the model uniformly; Image normalization, normalizing the pixel values of the images to a specific range; Image denoising: Using methods such as filtering to remove noise in the images; Image enhancement, further performing enhancement operations on the images, such as adjusting parameters such as brightness, contrast, and saturation of the images, to improve the quality of the images, enhance the distinguishability of features, enable the model to better learn the important features in the images, and improve the adaptability to different lighting conditions and image qualities.

[0032] The pre-trained image recognition model is obtained by training a preset deep learning object detection model with fish images with annotation information, specifically including: Obtain fish images annotated with fish species, body size, and swimming level information, and divide them into a training set, a validation set, and a test set according to a set ratio; Train the preset deep learning object detection model with the training set, and adjust the hyperparameters with the validation set to obtain the image recognition model; Test and evaluate the performance of the image recognition model with the test set and perform fine-tuning to obtain the final image recognition model.

[0033] In the present invention, data annotation, dataset division, and dataset construction are three key steps. First of all, data annotation is an important part of supervised learning. Professional tools (such as LabelImg) are used to annotate fish images, including information such as fish species, body size, and swimming level. When annotating, it is necessary to strictly follow the specifications to ensure that the annotation boxes are accurate and the attribute information is correct. To improve efficiency and quality, the annotated labels are checked and reviewed. The dataset is divided into a training set, a validation set, and a test set according to the ratio of 6:2:2. The training set is used for parameter learning, the validation set is used for adjusting hyperparameters, and the test set is used for final performance evaluation. Stratified sampling is adopted during division to ensure a reasonable distribution of each category, and a fixed random seed is set to ensure the consistency of the results. When organizing the dataset, the dataset is stored according to the requirements of model training, and label files and metadata information are generated to facilitate management and loading.

[0034] Step 200: Input the image to be recognized into a pre-trained image recognition model to generate visual perception data.

[0035] Specifically, input the image to be recognized into a pre-trained image recognition model; Through the image recognition model for feature fusion and target localization, determine the target species, body size, and swimming level, and obtain visual perception data through optimization by the total loss function.

[0036] In the present invention, through the recognition of fish species, fish body size, and the swimming level of fish by the image recognition model, a multi-dimensional comprehensive judgment of fish species is realized. This method overcomes the limitations of a single recognition method, can more accurately predict the mutual influence and spatial requirements between different fish, thereby effectively avoiding problems such as big fish eating small fish and swimming level conflicts, and significantly improving the scientific nature and success rate of the polyculture plan.

[0037] YOLO (You Only Look Once) is an advanced deep learning object detection algorithm first proposed by Joseph Redmon et al. in 2015. It is mainly used in the field of object detection and has the advantages of fast speed, high accuracy, strong interpretability, and wide applicability. It is one of the most important representatives in the current field of object detection. The structure of the YOLOv10 model includes a backbone network, a feature fusion part, and a detection head.

[0038] Based on the YOLOv10 model, the present invention constructs a multi-task learning framework. This framework shares the underlying feature extraction network to reduce waste of computing resources, and at the same time sets specific output layers for each task (species recognition, body size recognition, swimming level recognition) at the top layer to achieve parallel processing of multiple tasks.

[0039] The backbone structure of YOLO (You Only Look Once) is the fundamental part of the entire network, mainly responsible for extracting useful features from the input image. Through a series of operations such as convolutional layers and pooling layers, it gradually reduces the spatial resolution of the feature map while increasing the number of channels, thereby obtaining high-level semantic information of the image. To lightweight the network and facilitate integration into resource-constrained devices or application scenarios with high real-time requirements, the present invention uses GhostBottleneck to replace the backbone network of the YOLOv10 network, generating feature maps through inexpensive operations, reducing the computational amount while maintaining high performance. GhostBottleneck is the core module of GhostNet, consisting of two-stage GhostConv, achieving lightweight through feature reuse and linear transformation. The Conv module consists of three parts: Conv2d, Batch Normalization, and SiLU activation function module.

[0040] The image is input into the model, and the image size is (640×640×3).

[0041] After the first layer of Conv convolution, the dimension is reduced by a 3×3 convolution kernel to obtain the feature map F1, with a size of (320×320×64).

[0042] Immediately following is the GhostBottleneck module, which is composed of two Ghost modules connected in series. The first module serves as an expansion layer to expand the number of channels, and the second module reduces the number of channels. The Shortcut connection is between the inputs and outputs of the two Ghost modules, similar to the residual connection in ResNet. When the stride is 1, the module keeps the height and width of the input feature map unchanged for deepening the network; when the stride is 2, it is used to compress the height and width of the feature map. The structural diagram of this module is as Figure 3 shown. Input F1 into the first GhostBottleneck module with a stride of 2 to expand the output channels, obtaining F2, with a feature map size of (160×160×128).

[0043] Input F2 into the GhostBottleneck module with a stride of 1, and the output is F3, with a feature map size of (160×160×128).

[0044] Input F3 into the next GhostBottleneck module with a stride of 2, and the output is F4, with a feature map of (80×80×256).

[0045] Input F4 into the GhostBottleneck module with a stride of 1, and the output is F5, with a feature map size of (80×80×256).

[0046] Input F5 into the next GhostBottleneck module with a stride of 2, and the output is F6 with a feature map of (40×40×512).

[0047] Input F6 into the GhostBottleneck module with a stride of 1, and the output is F7 with a feature map size of (40×40×512).

[0048] Input F7 into the next GhostBottleneck module with a stride of 2, and the output is F8 with a feature map of (20×20×1024).

[0049] Input F8 into the GhostBottleneck module with a stride of 1, and the output is F9 with a feature map size of (20×20×1024).

[0050] Introduce three different attention mechanisms in the C2f structure during the feature fusion stage. For three different tasks, replace the C2f in the Neck network with the optimized C2f_Attn dynamic attention module, and the structure is as Figure 4 shown. Among them, the C2f_Attn module includes a first convolutional module connected in sequence, outputs the result to the next layer for the Spilt operation, splits it into three branches, and the three branches pass through the first convolutional module and the corresponding three attention mechanism modules (ECA, Coord, and PSA) in the second layer. The outputs of the above different branch feature extraction results are merged through the Concat operation according to different weights, and then go through the second convolutional module, and finally through the first convolutional module again.

[0051] Output F9 to the SPPF module. The SPPF module can extract feature information at different scales by performing max-pooling operations at multiple scales, and obtain F10 with a feature map size of (20×20×1024).

[0052] Output F10 to the next C2f_Attn module to obtain F11 with a feature map size of (20×20×1024).

[0053] F11 is upsampled, concatenated with F7, and output to the next C2f_Attn module to obtain F12 with a feature map size of (40×40×512).

[0054] F12 is output to the next upsampling, concatenated with F5, and output to the next C2f_Attn module to obtain the feature map F13 with a size of (80×80×256).

[0055] After passing through a layer of ordinary convolution, F13 is concatenated with F12 and output to the next C2f_Attn module, obtaining the feature map F14 with a size of (40×40×512).

[0056] F14 is output to the next SCDown module and concatenated with F11, then output to the next C2fCIB module, obtaining the feature map F15 with a size of (20×20×1024).

[0057] After feature extraction, object localization is performed. The Head part of YOLOv10 is a key component in its object detection process, responsible for final bounding box prediction, classification, and attribute judgment. Through innovative architecture design, the Head part of YOLOv10 realizes efficient object detection and multi-task recognition functions.

[0058] The multi-task processing framework of the present invention is for processing tasks such as fish species recognition, fish body shape recognition, and swimming level recognition, such as Figure 5 As shown, the three parallel tasks share the backbone network and the feature fusion module. After sharing the neck, a task-specific adapter is introduced, which includes 1x1 convolution + BN + ReLU to reduce interference between tasks.

[0059] For species recognition, after the feature fusion layer, the Head part of YOLOv10 outputs three feature maps of different scales. Each feature map corresponds to object detection tasks of different scales, respectively used to detect large, medium, and small objects. For each feature map, the detection head outputs the class confidence and the coordinates of the bounding box.

[0060] For body shape recognition, a body shape recognition branch is constructed, which consists of a three-level fully connected regression network combined with activation functions and regularization. Using the middle and high-level features of the Neck network in YOLOv10 as input, parameters such as body length and body width are output.

[0061] For swimming level recognition, a swimming level recognition branch is constructed. The dynamic region pooling method is used to evenly divide the feature map output by the Neck network into three segments in the y-axis direction, and the confidence and bounding box of the fish at the current level are obtained for the segmented feature map.

[0062] Visual perception data is obtained through optimization by the total loss function. Specifically, based on the multi-task requirements of YOLOv10, the loss functions of the three tasks of classification, regression, and swimming level analysis need to be jointly optimized. The total loss function can be defined as:

[0063] Among them, is the dynamic weight coefficient, which is adaptive through the GradNorm algorithm.

[0064] For the complex scenario of coexistence of multiple fish species, a multi-label classification loss is adopted:

[0065] where N represents the number of positive samples; C represents the total number of fish categories; represents the true label of the i-th sample belonging to category c; represents the probability that the model predicts the i-th sample belongs to category c.

[0066] Regression loss, for fish body size regression, the Smooth L1 loss function is adopted to enhance the robustness to outliers:

[0067] where N pos represents the number of valid positive samples; represents the body size parameter of the i-th sample; represents the body size parameter predicted by the model.

[0068] Swimming level loss, fusing the dual-path loss of swimming classification and bounding box regression:

[0069] Among them, represents the hierarchical classification cross-entropy loss, represents the bounding box regression loss.

[0070]

[0071] Among them, represents three region-level layers divided along the y-axis; represents that the i-th sample belongs to the k-th layer.

[0072]

[0073] Among them, is the bounding box regression loss function, represents the Euclidean distance between the center points of the predicted box and the true box; represents the diagonal length of the minimum bounding rectangle; represents the aspect ratio consistency parameter, is the dynamic weight coefficient.

[0074] F1-Score, Kappa coefficient, and IoU (Intersection over Union) are used as evaluation metrics; F1-Score is the harmonic mean that comprehensively measures Precision and Recall in classification tasks and is particularly suitable for scenarios with class imbalance.

[0075] 。

[0076] Among them, TP represents the number of correctly predicted positive samples; FP represents the number of incorrectly predicted positive samples; FN represents the number of incorrectly predicted negative samples.

[0077] The Kappa coefficient is used to evaluate the difference between the prediction results of a classification model and random guessing, taking into account the potential impact of class distribution, and is commonly used in inter-rater agreement tests.

[0078]

[0079] Among them, p0 represents the observed agreement rate; p e represents the expected agreement rate.

[0080] IoU (Intersection over Union) measures the overlap between the predicted region and the ground truth region and is widely used in object detection and image segmentation tasks; 。

[0081] The pre-trained knowledge graph matching model is obtained by training a preset deep feature matching model with a dataset with the corresponding relationship between the comprehensive feature vector and the knowledge graph, specifically including: Obtain fish breeding data and perform data cleaning and semantic alignment to obtain structured data and unstructured data; Based on the structured data and unstructured data, pre-define the core entity and relationship system through ontology, and combine rule-driven and deep learning to realize joint entity relationship extraction to construct a knowledge graph; Cross-modal associate the comprehensive feature vector with the knowledge graph, and train a preset deep feature matching model to obtain a knowledge graph matching model.

[0082] In the present invention, the construction of the knowledge graph is a systematic, multi-stage integration project, covering the entire process of data integration, knowledge modeling, storage optimization, dynamic update and intelligent application. The construction of the knowledge graph of domestic fish requires the construction of a vertical knowledge system around ornamental fish varieties, feeding management and disease prevention. In the data collection stage, relevant data such as variety characteristics (body shape, body color, fin shape), feeding parameters (water quality, water temperature, pH value, mixed breeding compatibility), disease symptoms (white spot disease, fin rot disease), feeding habits (carnivorous, herbivorous, omnivorous), nutritional requirements, water layer distribution, personality, etc. are integrated, and structured data and unstructured data that meet the requirements are collected by integrating existing databases, crawlers and other technologies. For example, a database contains cross-platform data of 1,200+ ornamental fish varieties. ETL tools and crawler technology are used to clean data and align semantics to solve noise filtering and entity synonym mapping. In the knowledge modeling stage, based on the ontology pre-defined core entities (such as FishSpecies, Disease) and relationship systems (compatibility, conflict, environmental dependence), the entity relationship joint extraction is realized by combining rule-driven (regular expression) and deep learning (BERT-BiLSTM+CRF model), and cross-modal image features are associated. The storage layer uses the Neo4j graph database, and the node attributes (such as aggressiveness score, temperature threshold) and dynamic relationships are designed through the Cypher language, and the query efficiency is improved by combining index optimization and sharding strategies. The dynamic update mechanism introduces event-driven architecture and graph neural network, and realizes knowledge self-evolution through crowdsourcing verification and version management. The application layer relies on the real-time reasoning engine to support mixed breeding conflict detection and water quality control suggestions, and realizes visual interaction with the help of AntV G6. The technology ecosystem integrates Scrapy, Apache NiFi, BERT and other tool chains to form a "data-model-storage-application" closed loop. In the future, causal reasoning and federated learning can be expanded to promote the transformation of knowledge graphs from static libraries to self-evolving cognitive systems, providing precise support for scenarios such as breeding decisions and ecological protection.

[0083] Step 300: Perform weighted fusion on the visual perception data and the sensor data to form a comprehensive feature vector.

[0084] Specifically, the improved YOLOv10-based multi-task framework is used to identify visual perception data (including fish species, quantity, body shape, and swimming level) and sensor data. The features from different sources are weighted and fused according to the preset weight coefficients to form a comprehensive feature vector, which comprehensively and accurately reflects the characteristics of the fish in the fish tank and the environmental status.

[0085] Step 400: Input the comprehensive feature vector into the pre-trained knowledge graph matching model, calculate the matching degree between the comprehensive feature vector and the polyculture rule in the knowledge graph according to the dynamic rule matching mechanism, and determine whether polyculture is possible based on the matching degree.

[0086] Specifically, input the comprehensive feature vector into the knowledge graph matching model; Match the comprehensive feature vector with the knowledge graph of polyculture of domestic fish in the knowledge graph matching model; according to various eigenvalue in the comprehensive feature vector, locate the corresponding fish species nodes, feature nodes and associated polyculture rule nodes in the knowledge graph; According to the dynamic rule matching mechanism, calculate the matching degree between the comprehensive feature vector and the polyculture rules in the knowledge graph, and judge whether the domestic fish can be polycultured according to the matching degree result.

[0087] In the present invention, the comprehensive feature vector is input into the knowledge graph matching module and matched with the constructed knowledge graph of polyculture of domestic fish. According to various eigenvalue in the comprehensive feature vector, locate the corresponding fish species nodes, feature nodes and associated polyculture rule nodes in the knowledge graph. According to the dynamic rule matching mechanism, calculate the matching degree between the comprehensive feature vector and the polyculture rules in the knowledge graph, and judge whether the domestic fish can be polycultured according to the matching degree result. If the matching degree reaches the preset polyculture compatibility threshold, output the conclusion that polyculture is possible, and further provide polyculture suggestions, such as appropriate polyculture density, matters needing attention, etc.; if the matching degree is lower than the threshold, output the conclusion that polyculture is not possible, and explain the possible risks and reasons, providing a comprehensive and accurate basis for polyculture judgment for users, helping them scientifically manage the polyculture of domestic fish and improving the success rate of polyculture and the survival quality of fish.

[0088] The present invention provides a method for judging fish polyculture based on a lightweight multi-task image recognition network and a knowledge graph. The GhostNet network is introduced into the backbone network of the original YOLOv10, which not only lightens the backbone network but also facilitates future deployment on mobile devices or edge devices. A dynamic attention selection module is introduced into the Neck network. Among them, ECA focuses on key texture features (such as fish body markings and fin shapes) through lightweight channel attention, effectively enhancing the discrimination ability for fine-grained species differences; CoordAttention can accurately capture the spatial position relationship of the fish body contour and improve the measurement accuracy of the body shape; PSA fuses environmental cues such as water surface reflection and depth blur through multi-scale spatial pooling, strengthening the perception ability of the underwater three-dimensional space position and movement trajectory. A multi-task processing framework for fish species recognition, body shape recognition, and swimming level recognition is constructed. In order to dynamically balance the learning weights and gradient conflicts between tasks and enhance the feature sharing efficiency, the loss function is improved to solve problems such as data imbalance, sample quality differences, and environmental interference, thereby improving the comprehensive performance and generalization ability of the model. This method dynamically integrates fish species information, body shape parameters, and swimming level information through multi-task processing, and combines the structured characteristics of the knowledge graph to systematically analyze information such as the temperament, feeding habits, and required environment of fish, realizing real-time early warning of the compatibility of fish polyculture, and significantly improving the accuracy and efficiency of fish polyculture risk assessment.

[0089] Reference Figure 6 , the present invention also discloses a fish polyculture judgment system based on a lightweight multi-task image recognition network and a knowledge graph, including: A data acquisition module 110, configured to acquire image data and sensor data, and preprocess the image data to obtain an image to be recognized; A visual perception data generation module 120, configured to input the image to be recognized into a pre-trained image recognition model to generate visual perception data; A fusion module 130, configured to perform weighted fusion on the visual perception data and the sensor data to form a comprehensive feature vector; A matching module 140, configured to input the comprehensive feature vector into a pre-trained knowledge graph matching model, calculate the matching degree between the comprehensive feature vector and the polyculture rules in the knowledge graph according to the dynamic rule matching mechanism, and determine whether polyculture is possible based on the matching degree; Among them, the pre-trained image recognition model is obtained by training a preset deep learning object detection model with fish images with annotation information, and the pre-trained knowledge graph matching model is obtained by training a preset deep feature matching model with a data set with the corresponding relationship between the comprehensive feature vector and the knowledge graph.

[0090] Among them, preprocessing the image data to obtain an image to be recognized specifically includes: Obtaining images of multiple fish species in polyculture under different environments through an image acquisition device; Performing data cleaning, data filtering, and data repair on the images of multiple fish species in polyculture under different environments to obtain primary image data; Performing image format conversion, image denoising, image normalization, and image enhancement on the primary image data to obtain an image to be recognized.

[0091] The pre-trained image recognition model is obtained by training a preset deep learning object detection model with fish images with annotation information, specifically including: Obtaining fish images annotated with fish species, body size, and swimming level information, and dividing them into a training set, a validation set, and a test set according to a set ratio; Training a preset deep learning object detection model with the training set, and adjusting hyperparameters with the validation set to obtain an image recognition model; Testing and evaluating the performance of the image recognition model with the test set and performing fine-tuning to obtain the final image recognition model.

[0092] Inputting the image to be recognized into the pre-trained image recognition model to generate visual perception data, specifically including: Inputting the image to be recognized into the pre-trained image recognition model; Performing feature fusion and target localization through the image recognition model to determine the target species, body size, and swimming level, and obtaining visual perception data through optimization by the total loss function.

[0093] The pre-trained knowledge graph matching model is obtained by training a preset deep feature matching model with a data set with the corresponding relationship between the comprehensive feature vector and the knowledge graph, specifically including: Obtaining fish breeding data and performing data cleaning and semantic alignment to obtain structured data and unstructured data; Based on the structured data and unstructured data, predefining the core entity and relationship system through ontology, and combining rule-driven and deep learning to realize joint entity relationship extraction and construct a knowledge graph; Cross-modally associating the comprehensive feature vector with the knowledge graph to train a preset deep feature matching model to obtain a knowledge graph matching model.

[0094] Inputting the comprehensive feature vector into the pre-trained knowledge graph matching model, calculating the matching degree between the comprehensive feature vector and the polyculture rules in the knowledge graph according to the dynamic rule matching mechanism, and determining whether polyculture is possible according to the matching degree, specifically including: Inputting the comprehensive feature vector into the knowledge graph matching model; Match the comprehensive feature vector with the knowledge graph of polyculture of farmed fish in the knowledge graph matching model of fish polyculture; locate the corresponding fish species nodes, feature nodes, and associated polyculture rule nodes in the knowledge graph according to various feature values in the comprehensive feature vector; According to the dynamic rule matching mechanism, calculate the matching degree between the comprehensive feature vector and the polyculture rules in the knowledge graph, and judge whether the farmed fish can be polycultured according to the matching degree result.

[0095] Based on a fish polyculture judgment system based on a lightweight multi-task image recognition network and a knowledge graph provided by the present invention, by performing image recognition on fish, information such as the species, body size, and swimming level of the fish is obtained, combined with information such as the temperament, water temperature, and pH value of different fish stored in the fish farming knowledge graph, and it is determined whether different species of fish are suitable for polyculture through multi-modal data fusion; it can accurately judge whether different species of fish are suitable for polyculture, avoid problems such as big fish eating small fish or fry, and unsuitable living environments for fish, and improve the efficiency and survival rate of fish farming.

[0096] Figure 7 Illustrate a schematic physical structure diagram of an electronic device, such as Figure 7 shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730, and a communication bus 740. Among them, the processor 710, the communication interface 720, and the memory 730 complete mutual communication through the communication bus 740. The processor 710 can call the logical instructions in the memory 730 to execute a fish polyculture judgment method based on a lightweight multi-task image recognition network and a knowledge graph. The method includes: obtaining image data and sensor data, preprocessing the image data to obtain an image to be recognized; inputting the image to be recognized into a pre-trained image recognition model to generate visual perception data; performing weighted fusion on the visual perception data and the sensor data to form a comprehensive feature vector; inputting the comprehensive feature vector into a pre-trained knowledge graph matching model, and according to the dynamic rule matching mechanism, calculate the matching degree between the comprehensive feature vector and the polyculture rules in the knowledge graph, and determine whether polyculture is possible according to the matching degree; wherein, the pre-trained image recognition model is obtained by training a preset deep learning object detection model with fish images with annotation information, and the pre-trained knowledge graph matching model is obtained by training a preset deep feature matching model with a data set with the corresponding relationship between the comprehensive feature vector and the knowledge graph.

[0097] In addition, when the logical instructions in the above-mentioned memory 730 are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs.

[0098] On the other hand, the present invention also provides a computer program product. The computer program product includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the fish polyculture judgment method for a lightweight multi-task image recognition network and a knowledge graph provided by the above-mentioned various methods. The method includes: obtaining image data and sensor data, and preprocessing the image data to obtain an image to be recognized; inputting the image to be recognized into a pre-trained image recognition model to generate visual perception data; performing weighted fusion on the visual perception data and the sensor data to form a comprehensive feature vector; inputting the comprehensive feature vector into a pre-trained knowledge graph matching model, and calculating the matching degree between the comprehensive feature vector and the polyculture rules in the knowledge graph according to a dynamic rule matching mechanism, and determining whether polyculture is possible based on the matching degree; wherein, the pre-trained image recognition model is obtained by training a preset deep learning object detection model with fish images with annotation information, and the pre-trained knowledge graph matching model is obtained by training a preset deep feature matching model with a data set having a corresponding relationship between the comprehensive feature vector and the knowledge graph.

[0099] In another aspect, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the method for judging polyculture of fish in a lightweight multi-task image recognition network and a knowledge graph provided by the above-mentioned various methods. The method includes: acquiring image data and sensor data, preprocessing the image data to obtain an image to be recognized; inputting the image to be recognized into a pre-trained image recognition model to generate visual perception data; performing weighted fusion on the visual perception data and the sensor data to form a comprehensive feature vector; inputting the comprehensive feature vector into a pre-trained knowledge graph matching model, and calculating the matching degree between the comprehensive feature vector and the polyculture rules in the knowledge graph according to a dynamic rule matching mechanism, and determining whether polyculture is possible based on the matching degree; wherein, the pre-trained image recognition model is obtained by training a preset deep learning object detection model with fish images with annotation information, and the pre-trained knowledge graph matching model is obtained by training a preset deep feature matching model with a data set having a corresponding relationship between the comprehensive feature vector and the knowledge graph.

[0100] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without creative work.

[0101] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, also by hardware. Based on this understanding, the essence of the above technical solution, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disc, etc., and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0102] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for judging fish polyculture based on a lightweight multi-task image recognition network and a knowledge graph, characterized in that, Including: Obtain image data and sensor data, and preprocess the image data to obtain an image to be recognized; Input the image to be recognized into a pre-trained image recognition model to generate visual perception data; Perform weighted fusion on the visual perception data and the sensor data to form a comprehensive feature vector; Input the comprehensive feature vector into a pre-trained knowledge graph matching model, and calculate the matching degree between the comprehensive feature vector and the polyculture rules in the knowledge graph according to the dynamic rule matching mechanism, and determine whether polyculture is possible based on the matching degree; Among them, the pre-trained image recognition model is obtained by training a preset deep learning object detection model with fish images with annotation information, and the pre-trained knowledge graph matching model is obtained by training a preset deep feature matching model with a data set with the corresponding relationship between the comprehensive feature vector and the knowledge graph.

2. The method for judging polyculture of fish based on a lightweight multi-task image recognition network and a knowledge graph according to claim 1, wherein The preprocessing of the image data to obtain an image to be recognized specifically includes: Obtain images of multiple fish polyculture in different environments through an image acquisition device; Perform data cleaning, data filtering, and data repair on the images of multiple fish polyculture in different environments to obtain primary image data; Perform image format conversion, image denoising, image normalization, and image enhancement on the primary image data to obtain an image to be recognized.

3. The fish polyculture judgment method based on a lightweight multi-task image recognition network and a knowledge graph according to claim 1, wherein The pre-trained image recognition model is obtained by training a preset deep learning object detection model with fish images with annotation information, specifically including: Obtain fish images annotated with fish species, body size, and swimming level information, and divide them into a training set, a validation set, and a test set according to a set ratio; Train a preset deep learning object detection model through the training set, and adjust hyperparameters through the validation set to obtain an image recognition model; Test and evaluate the performance of the image recognition model through the test set and perform fine-tuning to obtain the final image recognition model; Among them, the image recognition model is constructed based on a lightweight convolutional neural network and a multi-task learning framework, sharing the underlying feature extraction network within the framework, and setting specific output layers for each task at the top layer, and completing lightweight multi-task parallel processing through feature reuse and linear transformation.

4. The method for judging fish polyculture based on a lightweight multi-task image recognition network and a knowledge graph according to claim 1, wherein, The inputting the image to be recognized into a pre-trained image recognition model to generate visual perception data specifically includes: Input the image to be recognized into a pre-trained image recognition model; Perform feature fusion and target localization through the image recognition model to determine the target species, body size, and swimming level, and obtain visual perception data after optimization by the total loss function.

5. The method for judging polyculture of fish based on a lightweight multi-task image recognition network and a knowledge graph according to claim 1, characterized in that, The pre-trained knowledge graph matching model is obtained by training a preset deep feature matching model with a data set with the corresponding relationship between the comprehensive feature vector and the knowledge graph, specifically including: Obtain fish breeding data and perform data cleaning and semantic alignment to obtain structured data and unstructured data; Based on the structured data and unstructured data, pre-define the core entity and relationship system through ontology, and combine rule-driven and deep learning to realize joint entity relationship extraction and construct a knowledge graph; Train a preset deep feature matching model with the cross-modal associated comprehensive feature vector of the knowledge graph to obtain a knowledge graph matching model.

6. The method for judging fish polyculture based on a lightweight multi-task image recognition network and a knowledge graph according to claim 1, characterized in that Input the comprehensive feature vector into a pre-trained knowledge graph matching model, and calculate the matching degree between the comprehensive feature vector and the polyculture rules in the knowledge graph according to the dynamic rule matching mechanism. Determine whether polyculture is possible based on the matching degree. Specifically, it includes: Input the comprehensive feature vector into the knowledge graph matching model; Match the comprehensive feature vector with the polyculture knowledge graph of cultured fish in the knowledge graph matching model; locate the corresponding fish species nodes, feature nodes, and associated polyculture rule nodes in the knowledge graph according to various feature values in the comprehensive feature vector; Calculate the matching degree between the comprehensive feature vector and the polyculture rules in the knowledge graph according to the dynamic rule matching mechanism, and judge whether the cultured fish can be polycultured based on the matching degree result.

7. A fish polyculture judgment system based on a lightweight multi-task image recognition network and a knowledge graph, characterized in that, It includes: A data acquisition module for acquiring image data and sensor data, and preprocessing the image data to obtain an image to be recognized; A visual perception data generation module for inputting the image to be recognized into a pre-trained image recognition model to generate visual perception data; A fusion module for performing weighted fusion on the visual perception data and the sensor data to form a comprehensive feature vector; A matching module for inputting the comprehensive feature vector into a pre-trained knowledge graph matching model, calculating the matching degree between the comprehensive feature vector and the polyculture rules in the knowledge graph according to the dynamic rule matching mechanism, and determining whether polyculture is possible based on the matching degree; Among them, the pre-trained image recognition model is obtained by training a preset deep learning object detection model with fish images with annotation information, and the pre-trained knowledge graph matching model is obtained by training a preset deep feature matching model with a data set with the corresponding relationship between the comprehensive feature vector and the knowledge graph.

8. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and running on the processor, wherein, When the processor executes the computer program, it implements the fish polyculture judgment method based on the lightweight multi-task image recognition network and the knowledge graph according to any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the fish polyculture judgment method based on the lightweight multi-task image recognition network and the knowledge graph according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the fish polyculture judgment method based on the lightweight multi-task image recognition network and the knowledge graph according to any one of claims 1 to 6.

Images