Biological category detection method and device, detection terminal and storage medium

Abstract

The embodiment of the invention at least provides a biological category detection method and device, a detection terminal and a storage medium. The biological category detection method is applied to a detection terminal, and comprises the following steps: determining a to-be-selected category of a target biological in an environment based on a collected infrared thermal image, and determining a first position range and a biological thermal distribution image of the target biological in the infrared thermal image; determining a biological visual image corresponding to the target organism in the acquired environment image according to the first position range; the environment image and the infrared thermal image carry feature information of the target organism at the same moment; determining a predicted category and a predicted heat distribution image of the target organism based on the to-be-selected category and the organism visual image; and verifying the predicted heat distribution image based on the biological heat distribution image, and outputting the predicted category as a target category under the condition that the verification is passed.

Description

Technical Field

[0001] The present application relates to, but is not limited to, the field of information processing technology, and in particular to a biological category detection method, device, detection terminal, and storage medium. Background Art

[0002] Current outdoor biodetection solutions rely on low power consumption, requiring sensors to sample at low frequencies to minimize power consumption. Furthermore, animals move quickly, making accurate identification impossible using time series models. Consequently, the collected data only contains one or two valid frames. Therefore, the challenge of combining multimodal data to accurately identify outdoor biomarkers within these low power requirements and scenarios remains unsolved. Summary of the Invention

[0003] In view of this, embodiments of the present application provide at least one biological category detection method, device, detection terminal, and storage medium.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] In a first aspect, an embodiment of the present application provides a method for detecting biological categories, which is applied to a detection terminal. The method includes: determining a candidate category of a target organism in an environment based on a collected infrared thermal image, and determining a first position range and a biological thermal distribution image of the target organism in the infrared thermal image; determining a biological visual image corresponding to the target organism in the collected environmental image based on the first position range; the environmental image and the infrared thermal image carry characteristic information of the target organism at the same time; determining a predicted category and a predicted thermal distribution image of the target organism based on the candidate category and the biological visual image; verifying the predicted thermal distribution image based on the biological thermal distribution image, and if the verification passes, outputting the predicted category as the target category.

[0006] In the second aspect, an embodiment of the present application provides a biological category detection device, which is applied to the detection terminal, and the device includes: a first determination module, which is used to determine the candidate category of the target organism in the environment based on the collected infrared thermal image, and determine the first position range and biological thermal distribution image of the target organism in the infrared thermal image; a second determination module, which is used to determine the biological visual image corresponding to the target organism in the collected environmental image according to the first position range; the environmental image and the infrared thermal image carry the characteristic information of the target organism at the same time; a third determination module, which is used to determine the predicted category and predicted thermal distribution image of the target organism based on the candidate category and the biological visual image; a verification module, which is used to verify the predicted thermal distribution image based on the biological thermal distribution image, and output the predicted category as the target category if the verification passes. In the third aspect, an embodiment of the present application provides a detection terminal, which includes a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the processor implements some or all of the steps in the above method when executing the program.

[0007] In a third aspect, an embodiment of the present application provides a detection terminal, which includes a processor, a memory, a first acquisition unit, and a second acquisition unit, wherein the memory stores a computer program that can be run on the processor; the processor executes part or all of the steps in the above method when executing the computer program; the first acquisition unit is used to acquire infrared thermal images; and the second acquisition unit is used to acquire environmental images.

[0008] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which implements some or all of the steps in the above method when executed by a processor.

[0009] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements some or all of the steps in the above method.

[0010] Technical Effect: Based on the acquired infrared thermal image, the candidate category of the target organism in the environment can be determined, as well as the first position range of the target category in the infrared thermal image and the biological thermal distribution image. In this way, the target organism in the environment can be preliminarily identified based on the infrared thermal image. The biological visual image corresponding to the target organism is determined in the acquired environmental image based on the first position range. The environmental image and the infrared thermal image carry the characteristic information of the target organism at the same time. Based on the candidate category and the biological visual image, the predicted category and predicted thermal distribution image of the target organism are determined. In this way, the category of the target organism can be predicted by combining the infrared thermal image and the environmental image. Compared with the existing solution that only uses a single modality for detection, the accuracy of detecting the category of the target organism in the environment is improved. The predicted thermal distribution image is verified based on the biological thermal distribution image, and if the verification passes, the predicted category is output as the target category. In this way, the multi-modal prediction results are verified, further improving the accuracy of the category of the detected target organism.

[0011] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and do not limit the technical solutions of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The drawings herein are incorporated into and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and, together with the specification, are used to illustrate the technical solutions of the present application.

[0013] Figure 1 A schematic diagram of the implementation process of a biological category detection method provided in an embodiment of the present application;

[0014] Figure 2 A schematic diagram of the implementation process of a biological category detection method provided in an embodiment of the present application;

[0015] Figure 3 A schematic diagram of the implementation process of a biological category detection method provided in an embodiment of the present application;

[0016] Figure 4 A schematic diagram of the implementation process of a biological category detection method provided in an embodiment of the present application;

[0017] Figure 5 A schematic diagram of the implementation process of a biological category detection method provided in an embodiment of the present application;

[0018] Figure 6 A schematic diagram of the implementation process of a biological category detection method provided in an embodiment of the present application;

[0019] Figure 7A diagram illustrating a framework for implementing biological category detection according to an embodiment of the present application;

[0020] Figure 8 A schematic diagram of the structure of a biological type detection device provided in an embodiment of the present application;

[0021] Figure 9 A schematic diagram of a hardware entity of a detection terminal provided in an embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions of this application are further elaborated in detail below with reference to the accompanying drawings and embodiments. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0023] In the following description, references to "some embodiments" describe a subset of all possible embodiments. However, it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict. The terms "first / second / third" are merely used to distinguish similar organisms and do not represent a specific ordering of the organisms. It is understood that the specific order or sequence of "first / second / third" may be interchanged where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0024] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing this application only and are not intended to limit this application.

[0025] Currently, in outdoor biological detection solutions, due to low power consumption requirements, the sensor sampling frequency is low, such as 5 images per second or even lower, to reduce power consumption. In addition, animals move quickly and cannot be accurately identified through timing models. The collected data only contains one or two frames of valid images. Therefore, how to combine multimodal data to accurately identify outdoor organisms under low power consumption requirements and the above scenarios has become a problem to be solved.

[0026] The present invention provides a method for detecting biological species, which can be executed by a processor of a detection terminal. The detection terminal can be a server, laptop, tablet computer, desktop computer, smart TV, set-top box, mobile device, or other device with data processing capabilities.

[0027] Figure 1This is a schematic diagram of a biological category detection method provided in an embodiment of the present application. The method can be executed by a processor of a detection terminal. Figure 1 As shown, the method includes the following steps S101 to S104, combining Figure 1 The following steps are explained.

[0028] Step S101: determining a candidate category of a target organism in an environment based on a collected infrared thermal image, and determining a first position range and a thermal distribution image of the target organism in the infrared thermal image.

[0029] In some embodiments, at least one acquisition unit is provided in the detection terminal, which may include an infrared thermal image acquisition unit (such as an infrared thermal imager), an infrared light image acquisition unit (such as a near-infrared camera), and a visible light image acquisition unit (camera, visible light camera, etc.).

[0030] In some embodiments, an infrared thermal image acquisition unit provided on the detection terminal captures an infrared thermal image of the environment surrounding the detection terminal. The infrared thermal image is a visual image generated by the infrared thermal image acquisition unit capturing infrared light emitted by organisms. The infrared thermal image can reflect the temperature range distribution of various organisms in the surrounding environment.

[0031] In some embodiments, after the infrared thermal image of the surrounding environment of the detection terminal is captured by the infrared thermal image acquisition unit, the infrared thermal image is preprocessed, including performing non-uniformity correction on the infrared thermal image to eliminate the fixed pattern noise of the infrared thermal image acquisition unit; and performing temperature mapping on the infrared thermal image to convert the temperature signal represented by the infrared thermal image into an actual temperature value.

[0032] In some embodiments, the target organism can be an animal, a plant, a bionic animal, a bionic plant, etc.

[0033] In some embodiments, the target organism may include at least one organism, and the category to be selected may include a category of at least one organism.

[0034] In some embodiments, based on the temperature range distribution in the pre-processed infrared thermal image, a category to be selected is matched from a biological database; wherein the biological database includes temperature ranges of multiple categories of organisms.

[0035] In some embodiments, feature extraction can also be performed on the preprocessed infrared thermal image to obtain the thermodynamic characteristics and morphological characteristics of the target organism, wherein the thermodynamic characteristics characterize the temperature range distribution of the target organism, and the morphological characteristics characterize the contour characteristics of the target organism (such as the number of limbs, contour aspect ratio, etc.). Based on the contour characteristics and temperature range distribution of the target organism, the temperature range of each part of the target organism is determined, and based on the temperature range of each part of the target organism, the candidate category of the target organism is matched from the biological database.

[0036] In some embodiments, at least one candidate category of the target organism is matched from the organism database based on the target organism's temperature range distribution and the ambient temperature. It is understood that the candidate categories may include a first candidate category and a second candidate category; the first candidate category of the target organism is determined based on the difference between the target organism's average temperature and the ambient temperature, and the second candidate category of the target organism is matched from the organism database based on the target organism's temperature range distribution.

[0037] If the difference is greater than or equal to the preset temperature, the first candidate category characterizing the target organism is a homeothermic animal; if the difference is less than the preset temperature, the first candidate category characterizing the target organism is a poikilothermic animal.

[0038] For example, the average temperature of the target organism is 35°C and the ambient temperature is 25°C, the difference is 18°C, and the preset temperature is 10°C, so the first candidate category of the target organism is determined to be a homeothermic animal; the morphological characteristics of the target organism include four limbs and a tail, wherein the temperature of the limbs and tail is lower than the temperature of the torso, and then the second candidate category of the target organism matched from the biological database includes dogs, cats, foxes, etc.

[0039] In some embodiments, the target organism includes at least one organism, the first position range includes a position range corresponding to each organism, and the organism thermal distribution image includes a thermal distribution image of each organism.

[0040] In some embodiments, the upper left and lower right coordinates of the target organism are obtained from the infrared thermal image based on the target organism's outline. A rectangular frame is generated based on the upper left and lower right coordinates, and the rectangular frame is used as the target organism's location range. Compared to the related art that uses a mask image as the location range, using the rectangular frame to identify the target organism's location range does not require higher computing power, and improves the generation rate in outdoor low-power scenarios.

[0041] In some embodiments, based on the temperature range distribution of the infrared thermal image, the temperature range distribution of the target organism in the first position range is determined, and at least two temperature regions are generated based on the temperature range distribution of the target organism. Each temperature region corresponds to a different temperature range, and each temperature region is marked with a different marking state to obtain a thermal distribution image corresponding to the target organism.

[0042] Exemplarily, the at least two temperature zones include a low temperature zone, a medium temperature zone, and a high temperature zone. The low temperature zone is marked in blue, the medium temperature zone is marked in green, and the high temperature zone is marked in red.

[0043] Step S102: determining a biological visual image corresponding to the target organism in the collected environmental image according to the first position range.

[0044] The environmental image and the infrared thermal image carry characteristic information of the target organism at the same moment.

[0045] In some embodiments, the environmental image may include a visible light image and an infrared light image, which are acquired based on at least one acquisition unit provided in the detection terminal.

[0046] The environmental image and infrared thermal image are collected simultaneously by multiple acquisition units. In other words, the environmental image and infrared thermal image are collected by multiple acquisition units at the same time and location of the environment surrounding the search unit. The coordinate system transformation relationship between the multiple acquisition units is pre-calibrated.

[0047] The first biological visual image may include a visible light visual image and an infrared light visual image of the target organism.

[0048] In some embodiments, based on the pre-calibrated conversion relationship between the coordinate systems of each acquisition unit, the same area as the first position range of the thermal imaging image is located from the environmental image, and the environmental image is cropped to obtain a biological visual image of the target organism in the environmental image corresponding to the first position range in the thermal imaging image.

[0049] Among them, based on the coordinate conversion relationship between the acquisition unit corresponding to the infrared thermal image and the acquisition unit corresponding to the infrared light image, the same area as the first position range is located from the infrared light image, and the infrared light image is cropped to obtain the infrared light visual image of the target organism.

[0050] Among them, based on the coordinate conversion relationship between the acquisition unit corresponding to the infrared thermal image and the acquisition unit corresponding to the visible light image, the same area as the first position range is located from the visible light image, and the visible light image is cropped to obtain a visible light visual image of the target organism.

[0051] Step S103: Determine the predicted category and predicted thermal distribution image of the target organism based on the candidate category and the biological visual image.

[0052] In some embodiments, feature extraction is first performed on the biological visual image to obtain the texture features, contour features and thermal distribution features of the target organism. Based on the correspondence between the texture features, contour features and thermal distribution features and the biological category, screening is performed from the candidate categories to obtain the predicted category of the target organism.

[0053] In some embodiments, a standard temperature range distribution of an organism corresponding to the predicted category is obtained from a biological database based on the predicted category; the standard temperature range distribution characterizes the temperature distribution of each part of the organism of the predicted category; the contours of each part of the target organism in the biological visual image are obtained, and the temperature range corresponding to the contours of each part of the target organism in the biological visual image is matched based on the standard temperature range distribution. With each pixel point in the biological visual image as a reference, a temperature value is assigned to each pixel point in the contour of each part, thereby obtaining a predicted thermal distribution image of the target organism; and the category of the target organism in the environmental organism is determined based on the predicted thermal distribution image.

[0054] Among them, based on the temperature range of each part in the standard temperature range distribution, the average temperature value of each part is determined, and each pixel point in the contour of each part of the biological visual image is filled with the corresponding average temperature value, so as to obtain the predicted thermal distribution image of the target organism.

[0055] Exemplarily, the predicted category includes dogs, and the standard temperature range distribution of dogs includes the temperature range of the limbs, the temperature range of the head (the head can be further divided into ears, face, etc.), the temperature range of the trunk (can be further divided into back, chest, etc.), and the temperature range of the tail, and the average temperature value of each part is determined; in the biological visual image, the average temperature value corresponding to each part is matched, and the temperature value is assigned to the pixel point of each part according to the average temperature value, thereby obtaining the predicted thermal distribution image of the dog.

[0056] Step S104: verifying the predicted thermal distribution image based on the biological thermal distribution image, and outputting the predicted category as the target category if the verification passes.

[0057] In some embodiments, feature regions are respectively performed on the biological thermal distribution image and the predicted thermal distribution image to obtain feature vectors of the biological thermal distribution image and feature vectors of the predicted thermal distribution image, and the feature vectors of the biological thermal distribution image and the feature vectors of the predicted thermal distribution image are compared to obtain the similarity between the two, and the output thermal distribution image is verified based on the similarity between the two.

[0058] Wherein, when the similarity between the feature vector of the biological thermal distribution image and the feature vector of the predicted thermal distribution image is greater than or equal to a preset similarity, it indicates that the predicted distribution image has passed the verification, and the predicted category is output as the target category.

[0059] If the similarity between the feature vector of the biological thermal distribution image and the feature vector of the predicted thermal distribution image is less than a preset similarity, it indicates that the predicted distribution image verification fails and the predicted category is inaccurate, and the predicted category is not output.

[0060] In some embodiments, when the predicted thermal distribution image passes verification, the infrared thermal image, environmental image, biological visual image, and predicted thermal distribution image are packaged and sent to the cloud.

[0061] Among them, the detection terminal is also equipped with a voice interaction unit. The user controls the voice interaction unit of the detection terminal in the cloud, sends voice information (such as warning information, etc.) to the target organisms in the surrounding environment, and obtains the sound emitted by the target organisms through the voice interaction unit of the detection terminal.

[0062] In an embodiment of the present application, based on the captured infrared thermal image, it is possible to determine the candidate category of the target organism in the environment, as well as the first position range of the target category in the infrared thermal image and the biological thermal distribution image. In this way, the target organism in the environment can be preliminarily identified based on the infrared thermal image. The biological visual image corresponding to the target organism is determined in the captured environmental image based on the first position range. The environmental image and the infrared thermal image carry characteristic information of the target organism at the same time. Based on the candidate category and the biological visual image, the predicted category and predicted thermal distribution image of the target organism are determined. In this way, the category of the target organism can be predicted by combining the infrared thermal image and the environmental image. Compared with the prior art solution that uses only a single modality for detection, the accuracy of detecting the category of the target organism in the environment is improved. The predicted thermal distribution image is verified based on the biological thermal distribution image, and if the verification passes, the predicted category is output as the target category. In this way, the multi-modal prediction results are verified, further improving the accuracy of the category of the detected target organism.

[0063] Figure 2 The present invention provides a method for detecting biological categories according to an embodiment of the present invention. The method can be executed by a processor of a detection terminal. Figure 1 , Figure 1 Step S103 in the above example can be updated to step S201 to step S203, which will be combined with Figure 2 The steps shown are explained.

[0064] Step S201: extract features from the biological visual image to obtain a biological visual feature map corresponding to the biological visual image.

[0065] In some embodiments, feature extraction can be performed on the biological visual image based on an image processing algorithm or a deep learning algorithm to obtain the biological visual feature map.

[0066] In some embodiments, the biological visual image includes at least one of the following: a visible light image and an infrared light image. For the visible light image, the texture features, color features, shape features, etc. of the target organism are extracted; for the infrared light image, the temperature distribution, hot spot position, thermal boundary, etc. of the target organism are extracted.

[0067] Exemplarily, for visible light images, the distribution frequency of RGB three-channel pixel values ​​in the visible light image is counted based on the color feature extraction algorithm to obtain color features; the joint distribution of grayscale of pixel pairs in the visible light image at specific directions (such as 0°, 45°, 90°, 135°) and distances is counted based on the texture feature extraction algorithm to generate texture features such as contrast, energy, entropy, and correlation; and the contour features and shape features in the visible light image are extracted based on the Canny edge detection algorithm.

[0068] Exemplarily, for infrared images, the temperature of the target organism in the infrared image is extracted based on a thermal statistics algorithm, and the average temperature, temperature variance, and temperature extremes are calculated to obtain temperature features. The high-temperature area in the infrared image is segmented using the Otsu threshold segmentation algorithm to obtain the position and area of ​​the hot spot; the edges of temperature mutations (such as the thermal boundary between animals and the environment) are extracted based on the Sobel operator or the Laplace operator to obtain thermal boundary features.

[0069] In some embodiments, when the biological visual image includes a visible light image and an infrared light image, the features extracted from the visible light image and the features extracted from the infrared light image are fused to obtain a biological visual feature map.

[0070] Step S202: input the candidate category and the biological visual feature map into a classification network to obtain a predicted category of the target organism.

[0071] In some embodiments, the classification network can be a trained neural network model.

[0072] The trained neural network model may be a trained dual-stream convolutional neural network (Dual-StreamCNN, DS-CNN) or a trained Transformer neural network (Transformer Neural Network) model.

[0073] In some embodiments, if the biological visual feature map includes a visual feature map corresponding to a visible light image and a visual feature map corresponding to an infrared light image, if the trained neural network model is a dual-stream convolutional neural network (Dual-StreamCNN, DS-CNN), then its structure may include an input branch, a fusion layer, and a classifier, wherein the input branch includes a visible light branch, an infrared light branch, and a selected category branch.

[0074] In some embodiments, the biological visual feature map and the candidate category are input into a trained first convolutional neural network to determine a predicted category from the candidate categories, including: based on the need for preprocessing before inputting the biological visual feature map and the candidate category into the trained first convolutional neural network, including converting the candidate category into a coding format or vector format, and normalizing the biological visual feature map, and then inputting the coding vector of the candidate category and the normalized biological visual feature map into the trained first convolutional neural network to obtain the predicted category. Among them, the visual feature map corresponding to the visible light image is subjected to feature extraction through the visible light branch to obtain texture features, color features, and shape features; the visual feature map corresponding to the infrared light image is subjected to feature extraction through the infrared light branch to obtain temperature distribution, hot spot location, and thermal boundary; the coding vector of the candidate category is processed by the candidate category branch into a vector of the same dimension as the biological visual feature map; then, the above features are spliced ​​along the dimension through the fusion layer, and the spliced ​​features are fused; the fused features are input into the trained classifier; the probability distribution of each candidate category is obtained, and the candidate category with the highest probability is determined as the predicted category.

[0075] Among them, the visible light branch is equipped with a trained feature extraction model (a 50-layer residual neural network (for example, Residual Neural Network 50, ResNet50), an efficient neural network (Efficient Neural Network, EfficientNet), etc.), which outputs a visible light feature vector. The infrared light branch is also equipped with a trained feature extraction model, which outputs an infrared light feature vector. The branch of the to-be-selected category is used to convert the coding vector of the to-be-selected category into a vector of the same dimension as the visible light feature vector and the infrared light feature vector.

[0076] Among them, a trained fusion model is set in the fusion layer, which is used to splice and fuse the coding vector, visible light feature vector and infrared light feature vector of the selected category to obtain a fusion vector.

[0077] The classifier is provided with a trained classification model for obtaining the predicted probability of each candidate category based on the fusion vector.

[0078] In some embodiments, the training process in the above-mentioned first convolutional neural network includes: obtaining a first training set, including multiple visible light visual feature images and infrared light visual feature images corresponding to each biological category in multiple biological categories, wherein the visible light visual feature images and infrared light visual feature images have been standardized, and the visible light visual feature images and infrared light visual feature images carry the labeled categories of the biological categories; also including a set of to-be-selected categories, the set of to-be-selected categories including the encoding vector of each to-be-selected category; inputting the multiple visible light visual feature images, the multiple infrared light visual feature images, and the encoding vector of the to-be-selected category into the first convolutional neural network to be trained to obtain the predicted category, and adjusting the model parameters of the first convolutional neural network to be trained based on the loss value between the predicted category and the labeled category until the convergence condition is met, and outputting the trained first convolutional neural network.

[0079] In some embodiments, the biological visual feature map and the category to be selected are input into a trained second convolutional neural network to predict the category of the target organism based on the trained second convolutional neural network to obtain a predicted category; including: preprocessing the biological visual feature map and the category to be selected to obtain a standardized biological visual feature map and a semantic feature vector of the category to be selected; inputting the standardized biological visual feature map and the semantic feature vector of the category to be selected into the trained second convolutional neural network to obtain a predicted category for the target organism.

[0080] Among them, the visual feature map corresponding to the visible light image is subjected to feature extraction by the visible light branch to obtain texture features, color features, and shape features; the visual feature map corresponding to the infrared light image is subjected to feature extraction by the infrared light branch to obtain temperature distribution, hot spot location, and thermal boundary; the semantic feature vector of the to-be-selected category is processed into a vector of the same dimension as the biological visual feature map based on the to-be-selected category branch; then it is fused through the fusion layer to obtain a fused feature vector; the classifier outputs the probabilities of all biological categories based on the fused feature vector, so as to determine the biological category with the highest probability among all biological categories as the predicted category.

[0081] In some embodiments, the training process of the above-mentioned second convolutional neural network includes: obtaining a second training set, including visual feature maps corresponding to visible light images and visual feature maps corresponding to infrared images of organisms corresponding to multiple biological categories; the visual feature maps corresponding to the visible light images and the visual feature maps corresponding to the infrared images are standardized; the visual feature maps corresponding to the visible light images and the visual feature maps corresponding to the infrared images respectively carry corresponding labeled categories; the second training set also includes semantic descriptions of categories to be selected; the second training set also includes semantic descriptions of all preset number of biological categories; the visual feature maps corresponding to the visible light images, the visual feature maps corresponding to the infrared images, the semantic descriptions of the categories to be selected, and the semantic descriptions of a preset number of biological categories are input into the second convolutional neural network to be trained to obtain predicted categories; based on the loss value between the predicted categories and the labeled categories, the model parameters of the second convolutional neural network to be trained are adjusted until the convergence conditions are met, and the trained second convolutional neural network is output.

[0082] Step S203: inputting the standard thermal distribution information corresponding to the to-be-selected category and the biological visual feature map into a thermal distribution prediction network to determine a predicted thermal distribution image of the target organism.

[0083] In some embodiments, the standard thermal distribution information can be a standard thermal distribution characteristic diagram of the organism corresponding to the selected category, or it can be a text description of each part of the organism corresponding to the selected category; it characterizes the temperature distribution of each part of the organism corresponding to the selected category. For example, if the organism corresponding to the selected category is a dog, the temperature distribution of the dog's limbs is 34.0-36.0℃, the temperature distribution of the torso is 38.0-39.0℃, and the temperature distribution of the tail is 30.0-34.0℃.

[0084] In some embodiments, the heat distribution prediction network may be a trained second neural network model, such as a Transformer Neural Network model or a Convolutional Neural Network (CNN).

[0085] In some embodiments, before inputting the standard thermal distribution information and the biological visual feature map into the thermal distribution prediction network, the standard thermal distribution information and the biological visual feature map need to be preprocessed, including extracting the biological visual feature vector of the biological visual feature map and the standard features of the standard thermal distribution information; inputting the biological visual features and the standard features into a trained second neural network model to fuse the biological visual features and the standard features to obtain fused features, and generating a predicted thermal distribution image of the target organism based on the fused features.

[0086] In some embodiments, the biological visual feature map includes a visual feature map corresponding to the visible light image and a visual feature map corresponding to the infrared light image, and the standard thermal distribution information includes a standard thermal distribution map of the organism corresponding to the selected category; feature extraction is performed on the visual feature map corresponding to the visible light image and the visual feature map corresponding to the infrared light image to obtain a visible light feature vector and an infrared light feature vector of the target organism, the visible light feature vector includes a contour feature vector and a texture feature vector of each part, and the infrared light feature vector includes a temperature distribution vector, a hot spot position vector, and a thermal boundary feature vector; feature extraction is performed on the standard thermal distribution map to obtain standard temperature distribution features of each part of the organism corresponding to the selected category; the above features are standardized, and a learning feature vector (for example, a mapping matrix between part and temperature) is generated based on the standard temperature distribution features of each part, the visible light feature vector and the infrared light feature vector are fused to obtain a fused feature vector, and a predicted thermal distribution image is generated based on the fused feature vector.

[0087] Among them, the visible light feature vector and the infrared light feature vector are learned and fused, and a predicted thermal distribution image is generated based on the fused feature vector; the method includes: first, feature splicing is performed to obtain a spliced ​​feature vector, then the weight of each feature is calculated, and fusion is performed based on the weight of each feature to obtain a fused feature; finally, based on a deconvolution network and bilinear interpolation, the image size of the input biological visual feature is gradually upsampled to generate a predicted thermal distribution image.

[0088] In some embodiments, the training process of the above-mentioned second neural network model includes: obtaining a third data set, including visual feature maps corresponding to visible light images and infrared images of multiple biological categories, as well as standard thermal distribution information of the target category and annotated thermal distribution images of the target category; wherein, the visual feature maps corresponding to the visible light images and the visual feature maps corresponding to the infrared images of each biological category carry the annotated category of the corresponding organism; the standard thermal distribution information carries the temperature range distribution of each part of the corresponding organism; the visual feature maps corresponding to the visible light images and the visual feature maps corresponding to the infrared images of multiple biological categories, as well as the standard thermal distribution information of the target category are input into the second neural network model to be trained to obtain a predicted thermal distribution image, and based on the loss value between the predicted thermal distribution image and the standard thermal distribution image, the model parameters of the second neural network model to be trained are adjusted until the convergence conditions are met, and the trained second neural network model is output.

[0089] In an embodiment of the present application, by extracting features from a biological visual image, a biological visual feature map corresponding to the biological visual image can be obtained; based on the candidate category and the biological visual feature map, the category of the target organism is predicted through a classification network to obtain a predicted category; based on the standard thermal distribution information of the predicted category and the biological visual feature map, a predicted thermal distribution image of the target organism can be obtained through a thermal distribution prediction network. In this way, by performing multimodal fusion of the biological visual feature map of the target organism and the thermal distribution information corresponding to the predicted category, a predicted thermal distribution image of the target organism can be obtained to detect the category of the target organism, thereby improving the accuracy of detection.

[0090] Figure 3 The present invention provides a method for detecting biological categories according to an embodiment of the present invention. The method can be executed by a processor of a detection terminal. Figure 2 , Figure 2 Step S203 in the above example can be updated to step S301 and step S302, which will be combined with Figure 3 The steps shown are explained.

[0091] Step S301: Determine the segmentation result of the target organism based on the biological visual feature map.

[0092] The segmentation result includes the segmented region of the target organism and subregions of each body part of the target organism in the segmented region.

[0093] In some embodiments, the biological visual feature map includes a visual feature map corresponding to a visible light image and a visual feature map corresponding to an infrared light image. Based on the fused feature map of the visual feature map corresponding to the visible light image and the visual feature map corresponding to the infrared light image, segmentation is performed through an image segmentation model, including segmenting the fused feature map based on an overall mask to obtain a segmented area of ​​the target organism; segmenting the segmented area of ​​the target organism based on a part mask to obtain sub-areas of each body part of the target organism in the segmented area.

[0094] Exemplarily, the overall segmentation mask is 1, and the target organism part in the fusion feature map is represented by the overall segmentation mask 1 to obtain the segmentation area of ​​the target organism; the background part is represented by mask 0 to obtain the segmentation area of ​​the background part in the fusion feature map; for each part in the segmentation area of ​​the target organism, it is represented based on the segmentation mask corresponding to each body part in the target organism; for example, the head area is represented by mask 2, the torso is represented by mask 3, and the limbs are represented by mask 4 to obtain the sub-area of ​​each body part in the segmentation area.

[0095] Step S302: fitting a predicted thermal distribution image of the target organism based on the sub-regions of each body part of the target organism in the segmented region and the standard thermal distribution information.

[0096] In some embodiments, a predicted thermal distribution image is generated through a regression model based on the segmented sub-regions of each body part of the target organism (such as the head, torso, and limbs) as spatial constraints and combined with the standard thermal distribution information of the corresponding parts (such as the temperature mean and range).

[0097] In some embodiments, the mask of each body part and the standard thermal distribution information are input into a regression model to extract the spatial feature vector (part shape, position) of the mask of each body part, and the standard thermal distribution information is encoded to obtain a learnable feature vector (a mapping matrix between part and temperature); the learnable feature vector and the spatial feature vector are spliced ​​and decoded to obtain a predicted thermal distribution image.

[0098] In this embodiment, the target organism's visual signature image is segmented to obtain a segmented region of the target organism and subregions of each body part of the target organism within the segmented region. A predicted thermal distribution image of the target organism is fitted based on the subregions of the segmented region and the standard thermal distribution information. This improves the accuracy of subsequent detection of the target organism's classification, compared to a global fitting approach, by fitting based on the target organism's parts.

[0099] Figure 4 The present invention provides a method for detecting biological categories according to an embodiment of the present invention. The method can be executed by a processor of a detection terminal. Figure 2 , the environmental image includes an infrared light image and a visible light image; the biological visual feature image includes a first biological image corresponding to the infrared light image and a second biological image corresponding to the visible light image; Figure 2 Step S201 in the above example can be updated to step S401 to step S404, which will be combined with Figure 4 The steps shown are explained.

[0100] Step S401: resize the first biological image and the second biological image to obtain the first biological image and the second biological image with unified sizes.

[0101] In some embodiments, resizing refers to maintaining the same spatial resolution between the first biological image and the second biological image by image scaling, cropping, interpolation, etc. For example, both images may be resized to have the same pixel matrix size.

[0102] In some embodiments, it is necessary to determine target sizes of the first biological image and the second biological image, wherein the target size may be the size of the first biological image, the size of the second biological image, or other preset sizes.

[0103] In some embodiments, a scaling direction and a scaling size for the first biometric image are determined based on the size of the first biometric image and the target size, wherein the scaling direction is reduction if the size of the first biometric image is larger than the target size, and the scaling direction is enlargement if the size of the first biometric image is smaller than the target size.

[0104] In some embodiments, a scaling direction and a scaling size for the second biometric image are determined based on the size of the second biometric image and the target size, wherein the scaling direction is reduction if the size of the second biometric image is larger than the target size, and the scaling direction is enlargement if the size of the second biometric image is smaller than the target size.

[0105] Step S402: extract features from the first biological image based on a first feature extraction network to obtain a first visual feature map.

[0106] In some embodiments, the first feature extraction network may be a trained neural network, such as a customized convolutional neural network (Customized CNN), a mobile neural network version 3 (MobileNetV3), or the like.

[0107] In some embodiments, the first biological image is preprocessed and input into a trained neural network to obtain the temperature distribution characteristics, hot spot location characteristics, thermal boundary characteristics, etc. of the target organism, as well as the feature map corresponding to each feature, wherein the feature map corresponding to each feature is in the same dimension, and after multiple feature maps are spliced ​​along the same dimension, the spliced ​​feature maps are weightedly fused to obtain the first visual feature map.

[0108] Step S403: extract features from the second biological image based on a second feature extraction network to obtain a second visual feature map.

[0109] In some embodiments, the second feature extraction network can be a trained neural network, such as a convolutional neural network (CNN), a residual neural network 50 (Residual Neural Network 50, ResNet50), etc.

[0110] In some embodiments, the second biological image is preprocessed and input into a trained neural network to obtain the texture features, edge features, shape features, color features, etc. of the target organism, as well as a feature map corresponding to each feature, wherein the feature map corresponding to each feature is in the same dimension, and after multiple feature maps are spliced ​​along the same dimension, the spliced ​​feature maps are weightedly fused to obtain a second visual feature map.

[0111] Step S404: Fusing the first visual feature map and the second visual feature map to obtain a biological visual feature map corresponding to the biological visual image.

[0112] In some embodiments, weights are generated for the first visual feature map and the second visual feature map, and the feature maps are weighted and summed based on the weights of each feature map to obtain a fused biological visual feature map. The weights of the first visual feature map and the weights of the second visual feature map can be pre-set or generated based on a neural network model.

[0113] In some embodiments, the first visual feature map and the second visual feature map may be spliced ​​by dimension to obtain a biological visual feature map.

[0114] In the embodiment of the present application, the efficiency of Houxi feature fusion is improved by unifying the sizes of the first and second biological images. A first visual feature map is obtained by extracting features from the first biological image, and a second visual feature map is obtained by extracting features from the second biological image. Thus, by fusing the first and second visual feature maps, a biological visual feature map corresponding to the biological visual image can be obtained. This improves fusion efficiency and the accuracy of the generated biological visual feature map.

[0115] Figure 5 The present invention provides a method for detecting biological categories according to an embodiment of the present invention. The method can be executed by a processor of a detection terminal. Figure 1 , Figure 1 Step S104 in the above example can be updated to step S501 to step S504, which will be combined with Figure 5 The steps shown are explained.

[0116] Step S501 : segmenting the biological thermal distribution image and the predicted thermal distribution image based on the sub-regions of each body part of the target biological in the segmented region.

[0117] In some embodiments, segmenting the region refers to dividing the entire thermal distribution image into multiple local regions, each region corresponding to a specific body part, such as the head, torso, limbs, etc. By segmenting the thermal distribution image by body part, the difference between the actual thermal distribution and the predicted thermal distribution can be more accurately compared.

[0118] In some embodiments, based on the above, a segmentation mask exists for the sub-region corresponding to each body part, and the segmented sub-region mask (such as the head mask, the trunk mask, and the limb mask) is multiplied element-by-element with the biological thermal distribution image to extract the biological thermal distribution area of ​​each part; the segmented sub-region mask (such as the head mask, the trunk mask, and the limb mask) is multiplied element-by-element with the predicted thermal distribution image to extract the predicted biological thermal distribution area of ​​each part.

[0119] Step S502: For each of the body parts, determine a first similarity between a first sub-distribution image of the body part in the biological thermal distribution image and a second sub-distribution image of the body part in the predicted thermal distribution image.

[0120] In some embodiments, a first Euclidean distance of the biological heat distribution area corresponding to the target body part and a second Euclidean distance of the predicted heat distribution area corresponding to the target body part are calculated, and a first similarity is determined based on the first Euclidean distance and the second Euclidean distance.

[0121] Exemplarily, a first Euclidean distance of a biological heat distribution area corresponding to the head is calculated, a second Euclidean distance of a predicted heat distribution area corresponding to the head is calculated, and the first Euclidean distance and the second Euclidean distance are compared to obtain a first similarity.

[0122] Step S503: Generate a target similarity between the biological thermal distribution image and the predicted thermal distribution image based on the first similarity of each body part.

[0123] In some embodiments, the target similarity can be an overall similarity indicator derived by weighted averaging or other aggregation of the first similarities of all body parts. For example, the similarities of each part can be combined using an average, weighted average, or minimum and maximum values ​​to reflect the consistency between the entire predicted thermal distribution image and the actual thermal distribution image. Weights can be set based on the importance of different body parts, for example, the head and torso may have higher weights.

[0124] In some embodiments, a first hash value of the biological thermal distribution image as a whole and a second hash value of the predicted thermal distribution image as a whole are calculated and compared to obtain a second similarity; a weighted sum is performed based on the first similarity and the second similarity to obtain a target similarity.

[0125] Step S504: Determine a verification result based on the target similarity and the similarity threshold.

[0126] In some embodiments, the target similarity is compared with a similarity threshold. If the target similarity is greater than or equal to the similarity threshold, the representation verification is passed and the predicted category is output as the target category. If the target similarity is less than the similarity threshold, the representation verification fails, which means that the predicted category is inaccurate.

[0127] In the embodiment of the present application, by segmenting the thermal distribution image according to body parts and calculating the similarity of each part, and then generating the overall similarity and verifying it, the accuracy and robustness of the prediction model can be effectively improved. In this way, the matching degree between the predicted thermal distribution image and the actual thermal distribution image can be accurately evaluated, thereby optimizing the model training effect and further improving the accuracy of animal and biological recognition.

[0128] Figure 6 This is a schematic diagram of a biological category detection method provided in an embodiment of the present application. The method can be executed by a processor of a detection terminal based on Figure 1 , Figure 1 Step S102 in the above example can be updated to step S601 and step S602, which will be combined with Figure 6 The steps shown are explained.

[0129] Step S601: Based on a pre-calibrated conversion relationship, convert the first position range of the target organism in the infrared thermal image into a second position range of the target organism in the environment image.

[0130] In some embodiments, the pre-calibrated transformation relationship refers to the pixel-by-pixel mapping relationship between the infrared thermal image and the visible light environment image, established through system calibration. This relationship is typically achieved by aligning and calculating the ratios between different imaging modes using a calibration plate or a target of known dimensions. For example, during the installation phase, the device can simultaneously capture an infrared image and a texture image of the same scene and use an algorithm to identify the locations of common feature points in both images to establish a spatial coordinate transformation model.

[0131] In some embodiments, the first position range can be the contour area of ​​the target organism, and the area is segmented to obtain a two-dimensional mask of the area; and the upper left corner coordinates and the lower right corner coordinates of the contour area are obtained to generate a bounding box, and the coordinates of all pixels are extracted from the mask or bounding box as the first position range to be converted. Based on a pre-calibrated conversion relationship, the coordinates of the first position range are converted to the coordinates of the second position range in the environmental image.

[0132] Step S602: intercepting the environment image using the second position range to obtain the biological visual image; the proportion of the target organism in the biological visual image is the same as the proportion of the target organism in the biological visual image.

[0133] In some embodiments, using the second position range for interception refers to cutting out a sub-image region containing the target organism in the environment image according to the above-mentioned mapped coordinate range.

[0134] In some embodiments, based on the rectangular frame corresponding to the second position range, all pixel areas corresponding to the rectangular frame are intercepted from the environment image to obtain the biological visual image. No scaling operation is performed during the interception process, and the size ratio of the object in the original image remains unchanged.

[0135] In this embodiment, a pre-calibrated conversion relationship allows the precise mapping of target location information from infrared thermal images to visible light images, thereby improving image capture accuracy. By utilizing the second position range to capture the environmental image while maintaining the consistent proportions of the target organism, the resulting visual image of the organism is ensured to have high fidelity, thereby enhancing the accuracy of animal identification and behavioral analysis.

[0136] In some embodiments, in the above step S101, if the verification fails, the following implementation process is included:

[0137] If the verification fails, the candidate category, the biological visual image and the biological thermal distribution image are uploaded to the cloud; the cloud is used to generate a fused feature map based on the biological texture image and the biological thermal distribution image; and the target category of the target object is generated based on the fused feature map and the candidate category.

[0138] In some embodiments, when the recognition model of the local device cannot accurately determine the category of the target object, the system will upload the relevant data to the cloud. Compared with the detection terminal, the cloud has sufficient computing resources, can use larger-scale neural network models, and has stronger generalization capabilities. The cloud is used to receive and process data from the terminal device, including biological visual images (such as visible light images, infrared images), biological thermal distribution images (such as infrared thermal images), and information on the selected category.

[0139] In some embodiments, a fused feature map is a highly discriminative image representation generated by extracting multi-dimensional features from images of different modalities (e.g., visible light images and infrared images). The cloud uses feature-level fusion to extract the animal's appearance features from the bio-texture image and the body temperature distribution features from the bio-thermal distribution image. These two sets of features are then combined to form a unified feature vector for subsequent classification.

[0140] In some embodiments, the target category is the species or type of the target object that was ultimately determined. If local recognition fails, the cloud regenerates the target category by performing a more precise match and comparison of the uploaded image and features, ensuring the reliability of the recognition result. For example, if a fox is mistakenly identified as a wolf locally, the cloud can verify and correct the misclassification using a higher-level deep learning model.

[0141] In the embodiment of this application, by introducing a cloud-based processing mechanism when verification fails, the powerful computing power of the cloud is used to deeply fuse and re-identify the image data. This can make up for the limitations of the local recognition model, thereby improving the accuracy and stability of the overall recognition, and thus being able to adapt to more complex application scenarios.

[0142] The following describes an exemplary application of a biological category detection method provided in an embodiment of the present application in a practical scenario.

[0143] Monitoring and collecting information on animal behavior, species, and numbers is crucial for wildlife research, ecological conservation, and animal husbandry. Existing animal monitoring equipment has numerous limitations, including a single power supply, making it difficult to maintain long-term outdoor use; inconvenient data transmission, making it impossible to transmit collected data to a designated location in real time; and poor image acquisition and animal recognition accuracy at night or in low light conditions. Therefore, a remote, all-weather, outdoor intelligent animal recognition and data collection device is needed to overcome these challenges.

[0144] The data acquisition device includes a power supply and charging module with a power switch that can power on and off the device system, effectively saving battery consumption. It is equipped with a Type-C charging socket, which is reversible and easy to use, without having to distinguish between positive and negative directions. It supports USB 3.1 functions, capable of transmitting 4K-level video at higher speeds; it also supports high-current charging of 3A and 5A, and can also meet reverse charging requirements. The device converts solar energy into electrical energy through batteries and solar panels, outputting DC power that is stored in the battery, providing long-term power support for the device and ensuring continuous and stable operation of the device outdoors.

[0145] The acquisition device includes a light sensor and nighttime operation module, which uses a photoresistor. This light-sensitive resistor exhibits high resistance in the absence of light and rapidly decreases in resistance when exposed to strong light. When there is no light at night or the light level is dimmed, the light board circuit turns on the infrared light. Based on the mapping between different light intensities and infrared light intensities, the infrared light is turned on at different power levels. The camera also activates night mode to ensure clear image capture even at night or in low-light environments.

[0146] The acquisition device includes an image acquisition and processing module. The camera's photosensitive circuit and control components acquire and digitize images, extracting, identifying, classifying, and comparing animal AI features from the captured images. The thermal imaging component uses an infrared detector to receive infrared radiation energy from people or animals in front of the camera, generating infrared thermal images and similarly extracting, identifying, classifying, and comparing animal AI features. Both communicate with Wi-Fi SD cards and IoT SIM cards via the Serial Peripheral Interface (SPI) protocol. The camera captures a texture image of the environment, while the thermal imaging component captures an infrared thermal image of the environment. The infrared thermal image and texture image are then fused to create a target image. The target image is then identified using a trained image recognition model to determine the type of organism in the environment. The trained image recognition model is based on images of different types of organisms and the body temperature data of each animal.

[0147] The data collection device includes a data transmission module and an IoT SIM card. This module transmits data collected by the camera and thermal imaging system, along with recognition results, to the cloud or a designated data center in real time, providing stable and efficient network access for the device and enabling data exchange and information sharing between devices. A Wi-Fi SD card transmits data collected by the camera and recognition results to a wireless network terminal in real time, significantly improving the device's data sharing capabilities and convenience.

[0148] Among them, the collection device includes: a sound collection and interaction module, a microphone is used to capture sound, and a speaker is used to transmit sound. The two work together to combine the sound and the real-time image of the camera, enriching the monitoring information.

[0149] The acquisition device includes a trigger and alarm module that determines an alarm strategy based on the currently identified biological category (e.g., different alarm strategies for animals of different protection levels). Alarm strategies include audible and visual alarms, and reporting to the cloud or to a terminal device equipped by staff. The PIR detects infrared light emitted by people or animals near the camera and converts it into a wake-up trigger signal that can be used to activate the system during low-power operation. When a security incident is detected, the alarm device promptly sounds an alarm to alert or repel any destructive behavior by humans or animals.

[0150] The data collection device includes a status indicator and mounting module. The indicator lights reflect the device's operating status. The device offers multiple mounting options, including straps to a tree and screws to a wall, making it easy to install and use in various environments.

[0151] Through the collaborative work of the above modules, this device realizes the function of remote all-weather outdoor animal identification and data collection, solving the problems of insufficient power supply, poor recognition effect at night, inconvenient data transmission and so on in the existing technology, and has good practicality and scalability.

[0152] In addition, the device can also be expanded according to actual application needs, such as adding a GPS positioning module, multi-language voice prompt function, remote control module, etc., to enhance its ability to adapt to complex environments.

[0153] To sum up, the remote all-weather outdoor intelligent animal identification and collection device provided by the present invention can not only realize efficient identification and data collection of animals, but also has a stable power supply system, reliable night working capability, powerful data transmission function and flexible installation method, which has significant technological progress and broad application prospects.

[0154] Figure 7 This is a diagram of an implementation framework for biological category detection provided in an embodiment of the present application, wherein 701 is an infrared light image, 702 is a visible light image, and 703 is a candidate category generated based on the infrared thermal image; wherein, feature extraction is performed on the infrared light image 701 based on the infrared light feature extraction network to obtain an infrared light feature map, feature extraction is performed on the visible light image 702 based on the visible light feature extraction network to obtain a visible light feature map, the infrared light feature map and the visible light feature map are fused to obtain a fused feature map, and the fused feature map and the candidate category 703 are input into the trained classification network to obtain a predicted category, the predicted category and the fused feature map are input into the temperature distribution network to obtain a temperature distribution map, and the temperature distribution map and the candidate category 703 are input into the verification module to verify the predicted category. If the verification passes, the predicted category is output as the target category.

[0155] Based on the foregoing embodiments, the embodiments of the present application provide a biological category detection device, which includes the various units included and the various modules included in each unit, and can be implemented by a processor in a detection terminal; of course, it can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA), etc.

[0156] Figure 8 A schematic diagram of the structure of a biological type detection device provided in an embodiment of the present application is shown in FIG. Figure 8As shown, the biological category detection device 800 includes: a first determination module 801, a second determination module 802, a third determination module 803, and a verification module 804, wherein: the first determination module 801 is used to determine the candidate category of the target organism in the environment based on the collected infrared thermal image, and determine the first position range and biological thermal distribution image of the target organism in the infrared thermal image; the second determination module 802 is used to determine the biological visual image corresponding to the target organism in the collected environmental image according to the first position range; the environmental image and the infrared thermal image carry characteristic information of the target organism at the same time; the third determination module 803 is used to determine the predicted category and predicted thermal distribution image of the target organism based on the candidate category and the biological visual image; the verification module 804 is used to verify the predicted thermal distribution image based on the biological thermal distribution image, and if the verification passes, output the predicted category as the target category.

[0157] In some embodiments, the third determination module 803 is further used to perform feature extraction on the biological visual image to obtain a biological visual feature map corresponding to the biological visual image; input the selected category and the biological visual feature map into the classification network to obtain the predicted category of the target organism; input the standard thermal distribution information corresponding to the predicted category and the biological visual feature map into the thermal distribution prediction network to determine the predicted thermal distribution image of the target organism.

[0158] In some embodiments, the third determination module 803 is also used to determine the segmentation result of the target organism based on the biological visual feature map; the segmentation result includes the segmented area of ​​the target organism and the sub-areas of each body part of the target organism in the segmented area; based on the sub-areas of each body part of the target organism in the segmented area and the standard thermal distribution information, the predicted thermal distribution image of the target organism is fitted.

[0159] In some embodiments, the environmental image includes an infrared light image and a visible light image; the biological visual feature image includes a first biological image corresponding to the infrared light image and a second biological image corresponding to the visible light image; the third determination module 803 is also used to perform size transformation on the first biological image and the second biological image to obtain a first biological image and a second biological image with unified size; perform feature extraction on the first biological image based on a first feature extraction network to obtain a first visual feature map; perform feature extraction on the second biological image based on a second feature extraction network to obtain a second visual feature map; and fuse the first visual feature map and the second visual feature map to obtain a biological visual feature map corresponding to the biological visual image.

[0160] In some embodiments, the verification module 804 is further used to segment the biological thermal distribution image and the predicted thermal distribution image based on the sub-areas of each body part in the segmented area; for each body part, determine the first similarity between the first sub-distribution image of the body part in the biological thermal distribution image and the second sub-distribution image of the body part in the predicted thermal distribution image; based on the first similarity of each body part, generate a target similarity between the biological thermal distribution image and the predicted thermal distribution image; and determine a verification result based on the target similarity and a similarity threshold.

[0161] In some embodiments, the first determination module 801 is further used to convert the first position range of the target organism in the infrared thermal image into the second position range of the target organism in the environmental image based on a pre-calibrated conversion relationship; use the second position range to intercept the environmental image to obtain the biological visual image; the proportion of the target organism in the biological visual image is the same as the proportion of the target organism in the biological visual image.

[0162] In some embodiments, the verification module 804 is further used to upload the candidate category, the biological visual image and the biological thermal distribution image to the cloud if the verification fails; the cloud is used to generate a fused feature map based on the biological texture image and the biological thermal distribution image; and generate a target category of the target object based on the fused feature map and the candidate category.

[0163] The description of the above device embodiment is similar to the description of the above method embodiment and has similar beneficial effects as the method embodiment. In some embodiments, the functions or modules included in the device provided in the embodiments of the present application can be used to perform the methods described in the above method embodiments. For technical details not disclosed in the device embodiments of the present application, please refer to the description of the method embodiments of the present application for understanding.

[0164] It should be noted that, in the embodiment of the present application, if the above method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a detection terminal (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific hardware, software or firmware, or any combination of hardware, software and firmware.

[0165] An embodiment of the present application provides a detection terminal, a processor, a memory, a first acquisition unit and a second acquisition unit, wherein the memory stores a computer program that can be run on the processor; when the processor executes the computer program, some or all of the steps in the above method are implemented; the first acquisition unit is used to acquire infrared thermal images; and the second acquisition unit is used to acquire environmental images.

[0166] The present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements some or all of the steps in the above method. The computer-readable storage medium may be transient or non-transient.

[0167] An embodiment of the present application provides a computer program, including computer-readable code. When the computer-readable code is run in a detection terminal, a processor in the detection terminal executes some or all of the steps for implementing the above method.

[0168] An embodiment of the present application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and when the computer program is read and executed by a computer, implements some or all of the steps in the above method. The computer program product can be implemented specifically by hardware, software, or a combination thereof. In some embodiments, the computer program product is embodied as a computer storage medium. In other embodiments, the computer program product is embodied as a software product, such as a software development kit (SDK), etc.

[0169] It should be noted that the descriptions of the various embodiments above tend to emphasize the differences between the various embodiments, and their similarities or similarities can be referenced to each other. The descriptions of the above device, storage medium, computer program, and computer program product embodiments are similar to the descriptions of the above method embodiments and have similar beneficial effects as the method embodiments. For technical details not disclosed in the embodiments of the device, storage medium, computer program, and computer program product of this application, please refer to the description of the method embodiments of this application for understanding.

[0170] Figure 9 A hardware entity diagram of a detection terminal provided in an embodiment of the present application is shown as follows: Figure 9 As shown, the hardware entity of the detection terminal 900 includes: a processor 901, a memory 902, a first acquisition unit 903, and a second acquisition unit 904, wherein the memory 902 stores a computer program that can be run on the processor 901, and when the processor 901 executes the program, the steps in the method of any of the above embodiments are implemented.

[0171] The memory 902 stores computer programs that can be run on the processor. The memory 902 is configured to store instructions and applications executable by the processor 901. It can also cache data to be processed or processed by the processor 901 and each module in the detection terminal 900 (for example, image data, audio data, voice communication data and video communication data), which can be implemented through flash memory (FLASH) or random access memory (RAM).

[0172] When the processor 901 executes the program, the steps of any of the above methods are implemented. The processor 901 generally controls the overall operation of the detection terminal 900.

[0173] The first acquisition unit 903 is used to acquire infrared thermal images.

[0174] The second acquisition unit 904 is used to acquire an environment image.

[0175] An embodiment of the present application provides a computer storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement the steps of the method of any of the above embodiments.

[0176] It should be noted that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0177] The processor may be at least one of an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a central processing unit (CPU), a controller, a microcontroller, and a microprocessor. It is understood that the electronic device that implements the functions of the processor may also be other electronic devices, which are not specifically limited in the embodiments of the present application.

[0178] The above-mentioned computer storage medium / memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory (Flash Memory), a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); it can also be various terminals including one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc.

[0179] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned steps / processes does not mean the order of execution, and the execution order of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The above-mentioned serial numbers of the embodiments of the present application are for description only and do not represent the advantages and disadvantages of the embodiments.

[0180] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0181] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0182] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed across multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the scheme of this embodiment.

[0183] In addition, the functional units in the embodiments of the present application can all be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the above-mentioned integrated unit can be implemented in the form of hardware or in the form of hardware plus software functional units. It can be understood by ordinary technicians in this field that all or part of the steps of the above-mentioned method embodiments can be completed by hardware related to program instructions, and the above-mentioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiments; and the above-mentioned storage medium includes: various media that can store program codes, such as mobile storage devices, read-only memories (ROMs), magnetic disks or optical disks.

[0184] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the relevant technology, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a detection terminal (which can be a personal computer, server, or network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0185] The above is only an implementation method of the present application, but the scope of protection of the present application is not limited thereto. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed in this application, which should be covered by the scope of protection of the present application.

Claims

1. A method for detecting biological categories, characterized in that: Applied to a detection terminal, the method includes: Determining a candidate category of a target organism in the environment based on the collected infrared thermal image, and determining a first position range and a thermal distribution image of the target organism in the infrared thermal image; Determining a biological visual image corresponding to the target organism in the collected environmental image according to the first position range; the environmental image and the infrared thermal image carry characteristic information of the target organism at the same time; Determining a predicted category and a predicted thermal distribution image of the target organism based on the candidate category and the biological visual image; The predicted thermal distribution image is verified based on the biological thermal distribution image, and if the verification passes, the predicted category is output as the target category.

2. The method according to claim 1, characterized in that The step of determining the predicted category and the predicted thermal distribution image of the target organism based on the candidate category and the biological visual image includes: Extracting features from the biological visual image to obtain a biological visual feature map corresponding to the biological visual image; Inputting the candidate category and the biological visual feature map into a classification network to obtain a predicted category of the target organism; The standard thermal distribution information corresponding to the predicted category and the biological visual feature map are input into a thermal distribution prediction network to determine a predicted thermal distribution image of the target organism.

3. The method according to claim 2, characterized in that The step of inputting the standard thermal distribution information corresponding to the predicted category and the biological visual feature map into a thermal distribution prediction network to determine a predicted thermal distribution image of the target organism includes: Determining a segmentation result of the target organism based on the biological visual feature map; the segmentation result includes a segmented region of the target organism and subregions of each body part of the target organism within the segmented region; Based on the sub-regions of the segmented region and the standard thermal distribution information of each body part of the target organism, a predicted thermal distribution image of the target organism is fitted.

4. The method according to claim 2, characterized in that The environmental image includes an infrared light image and a visible light image; the biological visual feature image includes a first biological image corresponding to the infrared light image and a second biological image corresponding to the visible light image; and the feature extraction of the biological visual image to obtain a biological visual feature map corresponding to the biological visual image includes: performing size transformation on the first biological image and the second biological image to obtain the first biological image and the second biological image with unified sizes; performing feature extraction on the first biological image based on a first feature extraction network to obtain a first visual feature map; performing feature extraction on the second biological image based on a second feature extraction network to obtain a second visual feature map; The first visual feature map and the second visual feature map are fused to obtain a biological visual feature map corresponding to the biological visual image.

5. The method according to claim 3, characterized in that The verifying the predicted thermal distribution image based on the biological thermal distribution image includes: segmenting the biological thermal distribution image and the predicted thermal distribution image based on the sub-regions of the segmented region for each body part of the target biological; For each of the body parts, determining a first similarity between a first sub-distribution image of the body part in the biological thermal distribution image and a second sub-distribution image of the body part in the predicted thermal distribution image; generating a target similarity between the biological thermal distribution image and the predicted thermal distribution image based on the first similarity of each of the body parts; A verification result is determined based on the target similarity and the similarity threshold.

6. The method according to any one of claims 1 to 5, characterized in that The determining of the biological visual image corresponding to the target organism in the collected environmental image according to the first position range includes: Based on a pre-calibrated conversion relationship, converting the first position range of the target organism in the infrared thermal image into a second position range of the target organism in the environment image; The environmental image is intercepted using the second position range to obtain the biological visual image; the proportion of the target organism in the biological visual image is the same as the proportion of the target organism in the biological visual image.

7. The method according to any one of claims 1 to 5, characterized in that The method further comprises: If the verification fails, the candidate category, the biological visual image and the biological thermal distribution image are uploaded to the cloud; the cloud is used to generate a fused feature map based on the biological texture image and the biological thermal distribution image; and the target category of the target object is generated based on the fused feature map and the candidate category.

8. A biological type detection device, characterized in that: Applied to the detection terminal, the device includes: A first determination module is configured to determine a candidate category of a target organism in an environment based on the acquired infrared thermal image, and to determine a first position range and a thermal distribution image of the target organism in the infrared thermal image; A second determining module is configured to determine a biological visual image corresponding to the target organism in the collected environmental image according to the first position range; the environmental image and the infrared thermal image carry characteristic information of the target organism at the same time; a third determination module, configured to determine a predicted category and a predicted thermal distribution image of the target organism based on the candidate category and the biological visual image; The verification module is used to verify the predicted thermal distribution image based on the biological thermal distribution image, and output the predicted category as the target category if the verification passes.

9. A detection terminal, characterized in that: The detection terminal includes a processor, a memory, a first acquisition unit and a second acquisition unit, wherein: The memory stores a computer program executable on the processor; When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented; The first acquisition unit is used to acquire infrared thermal images; The second acquisition unit is used to acquire an environment image.

10. A computer-readable storage medium, characterized in that A computer program is stored thereon, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images