Cross-modal livestock behavior retrieval method and device

Abstract

The invention provides a cross-modal livestock behavior retrieval method and device, and relates to the technical field of text-video retrieval. The method comprises the following steps: acquiring a livestock video and an input text input by a user, wherein the input text comprises a livestock behavior event; extracting a target sentence and nouns and verbs of the target sentence in the input text; based on a word set corresponding to the livestock behavior retrieval tag, determining a retrieval tag of a noun and a retrieval tag of a verb of the target sentence; searching video features matched with the retrieval tag of the noun, the retrieval tag of the verb and the sentence structure of the target sentence from the video-text multi-channel feature set; and based on the matched video features, retrieving livestock behavior events in the livestock video. Based on the scheme, the accuracy of retrieving the livestock behavior event from the livestock video based on the input text can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of text-video retrieval, and in particular to a cross-modal livestock behavior retrieval method and device. Background Art

[0002] With the development of image retrieval technology, image retrieval technology has gradually begun to be applied in the field of animal husbandry, so the accuracy of image retrieval is crucial.

[0003] Generally, animal behavior can reflect the physiological health of animals. By timely identifying abnormal animal behavior, the spread of diseases can be prevented, feeding methods can be adjusted, and breeding can be monitored. At present, farmers can use cameras to record videos and images of animals. However, the content of videos and images is rich, but the amount of livestock information is large. If farmers need to quickly retrieve the video segments of livestock behaviors that farmers are concerned about from the massive livestock breeding monitoring videos and locate specific livestock individuals with specific behaviors, manual query has a high retrieval accuracy of animal behavior, but the efficiency is low.

[0004] Traditional methods can learn a shared embedding space to establish a connection between video and text. The main processing method is to directly retrieve the target video based on the input text. Due to the diversity of text descriptions input by different users and the poor semantic consistency between different modalities, the current retrieval results are currently inaccurate. For example, when searching for a cow drinking water on the ground, the retrieval result may be a video of a cow standing by the water. Summary of the invention

[0005] The purpose of the present invention is to provide a cross-modal livestock behavior retrieval method and device to address the problem of poor accuracy of retrieval results in the prior art, which can improve the accuracy of retrieving livestock of target livestock behavior events from livestock videos based on input text.

[0006] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0007] In a first aspect, the present invention provides a cross-modal livestock behavior retrieval method, which obtains livestock videos and input text input by users, wherein the input text includes livestock behavior events; extracts target sentences, nouns and verbs in the input text; determines the retrieval tags of nouns and verbs in the target sentence based on the word set corresponding to the livestock behavior retrieval tag; searches for video features that match the retrieval tags of nouns, the retrieval tags of verbs and the sentence structure of the target sentence from a video-text multi-channel feature set; retrieves livestock behavior events in livestock videos based on the matched video features; wherein the word set corresponding to the livestock behavior retrieval tag includes: a synonym set of the noun retrieval tag, a synonym set of the verb retrieval tag; and the video-text multi-channel feature is a fusion feature obtained by performing temporal semantic enhancement and content semantic enhancement on both the description text and the video corresponding to the livestock behavior event.

[0008] In a second aspect, an embodiment of the present invention provides a cross-modal livestock behavior retrieval device, which includes: a data acquisition module, a feature extraction module, a retrieval label determination module, a feature matching module and a retrieval module; the data acquisition module is used to acquire livestock videos and input text input by a user, the input text includes livestock behavior events; the feature extraction module is used to extract target sentences, nouns and verbs in the input text; the retrieval label determination module is used to determine the retrieval label of the noun and the retrieval label of the verb of the target sentence based on the word set corresponding to the livestock behavior retrieval label; the feature extraction module is used to search for video features that match the retrieval label of the noun, the retrieval label of the verb and the sentence structure of the target sentence from the video-text multi-channel feature set; the feature matching module is used to retrieve livestock behavior events in livestock videos based on the matched video features; wherein the word set corresponding to the livestock behavior retrieval label includes: a synonym set of the retrieval label of the noun, and a synonym set of the retrieval label of the verb; the video-text multi-channel feature is a fusion feature after both the description text and the video corresponding to the livestock behavior event are subjected to temporal semantic enhancement and content semantic enhancement.

[0009] According to a third aspect of an embodiment of the present invention, a computer device is provided, comprising: a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any of the above cross-modal livestock behavior retrieval methods when executing the computer program.

[0010] A fourth aspect of the embodiments of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above cross-modal livestock behavior retrieval methods.

[0011] The cross-modal livestock behavior retrieval method provided by the embodiment of the present invention, in the first aspect, by performing content semantic enhancement and time semantic enhancement on livestock videos, the essential semantic information of livestock videos can be enriched, and by performing content semantic enhancement and time semantic enhancement on livestock input texts, the essential semantic information of input texts can be enriched, thereby obtaining the essential semantic information of the entire event under different modalities; in the second aspect, based on the features of livestock videos after semantic enhancement and the features of input texts after semantic enhancement, hierarchical fusion is performed into video multi-channel features and text multi-channel features. Through hierarchical fusion, the complementary features between the two different modalities of video and text can be combined, the redundant information of the fused features can be reduced, a more comprehensive understanding of the entire livestock behavior event can be obtained, and the semantic consistency between different modalities can be enhanced. The accuracy of livestock video-text retrieval can be effectively improved, the cost of human control can be reduced, and precise and intelligent real-time monitoring and management of a large number of livestock behaviors can be realized. For example, livestock behavior videos that users are concerned about can be accurately and quickly retrieved from massive agricultural livestock monitoring videos according to the input text entered by the user, and livestock with abnormal behaviors can be located. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 One of the schematic diagrams of the model training process of the cross-modal livestock behavior retrieval method provided in an embodiment of the present invention.

[0013] Figure 2 The second schematic diagram of the model training process of the cross-modal livestock behavior retrieval method provided in an embodiment of the present invention.

[0014] Figure 3 A semantically enhanced logic schematic diagram provided by an embodiment of the present invention.

[0015] Figure 4 A schematic diagram of a cross-modal livestock behavior retrieval training model provided in an embodiment of the present invention.

[0016] Figure 5 A cross-modal livestock behavior retrieval training logic schematic diagram provided by an embodiment of the present invention.

[0017] Figure 6 A schematic flow chart of a cross-modal livestock behavior retrieval method provided in an embodiment of the present invention.

[0018] Figure 7 A hardware structure diagram of a computer device in which a cross-modal livestock behavior retrieval system is provided in an embodiment of the present invention.

[0019] Figure 8 A schematic diagram of the structure of a cross-modal livestock behavior retrieval device is provided for an embodiment of the present invention. DETAILED DESCRIPTION

[0020] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.

[0021] The terms used in the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The singular forms "a", "the" and "the" used in the present invention and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0022] It should be understood that although the terms first, second, third, etc. may be used in the present invention to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining."

[0023] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0024] First, the model training process of the cross-modal livestock retrieval method provided by an embodiment of the present invention is given.

[0025] Figure 1 A schematic diagram of a model training process of a cross-modal livestock behavior retrieval method provided by an embodiment of the present invention, such as Figure 1 As shown in , the method includes the following S101 to S105:

[0026] S101, obtaining a training video and a description text matching the livestock behavior events in the training video.

[0027] The training video is any video in the training video set.

[0028] Specifically, usually, livestock behavior events refer to specific behaviors or activities related to farm animals or livestock (such as cattle, sheep, pigs, etc.). Livestock behavior events can be livestock behaviors that last for a period of time, livestock behaviors that last for a long time, or livestock behaviors that last for a short time, such as feeding behaviors, breeding behaviors, social behaviors, alertness and escape behaviors, migration and foraging behaviors, disease or discomfort behaviors, training and obedience behaviors, etc., which are not specifically limited in this embodiment of the present invention.

[0029] It is understandable that the description text is a manually input text description that matches the livestock behavior event in the training video. It is understandable that the description texts input by different users are diverse and different.

[0030] For example, for a cow that is ruminating, the description text entered by user 1 is "a cow that is ruminating", the description text entered by user 2 is "a cow that chews for a long time and does not gnaw on the grass", the description text entered by user 3 is "the cow is ruminating", and the description text entered by user 4 is "the cow keeps chewing."

[0031] S102: Perform content semantic enhancement on the training video and the description text, and perform time semantic enhancement on the training video and the description text to obtain semantically enhanced video features and semantically enhanced text features.

[0032] Typically, the content semantics of livestock behavior events involve the behavior patterns, habits, and interactions of livestock (such as cows, sheep, pigs, etc.).

[0033] Exemplarily, the content semantics describing the livestock behavior event in the training video may include at least one of the following information: the type of livestock object where the livestock behavior event occurred, the specific object individual, the location where the event occurred, and the specific behavior action of the livestock.

[0034] It can be understood that the temporal semantics of livestock behavior events involves the behavior patterns, habits, and relationships between livestock and the environment in a specific time period, such as behavior timing, seasonal changes, growth and development stages, environmental influences, event sequences, and temporal regularities.

[0035] Exemplarily, the temporal semantics describing livestock behavior events in the training video may include at least one of the following information: the start time of the livestock behavior event (starting video frame), the duration of the behavior event (the continuous video frames during which the behavior event lasts), and the end time of the behavior event (ending video frame).

[0036] For example, the content semantics describing the livestock behavior event in the description text may include at least one of the following information: a sentence description of the livestock behavior event, a livestock object description, a livestock action description, and a location description.

[0037] For example, the temporal semantics of the livestock behavior event described in the description text may include at least one of the following information: tense information of the verb of the livestock behavior event, time information contained in the verb, and time information contained in the noun.

[0038] It can be understood that in the embodiment of the present invention, cross-modal semantic enhancement can be performed, that is, both the training video and the description text are semantically enhanced, and the cross-modal semantic enhancement can obtain the essential semantic information of the entire event of livestock behavior.

[0039] By enhancing the temporal semantics of livestock behavior events in training videos, the essential semantic information of the entire time of livestock behavior can be obtained.

[0040] For example, the essential semantics of cattle's rumination behavior is that the food in the cow's stomach returns to the cow's mouth through vomiting and then chewing again. This behavior can continue for a period of time. Therefore, by enhancing the temporal semantics of describing livestock behavior events in training videos, the cattle's rumination behavior can be accurately described instead of chewing grass.

[0041] Specifically, content semantic enhancement and temporal semantic enhancement are performed on both the training video and the text, thereby improving the consistency between the content and key temporal information semantics of the livestock behavior events presented in the training video and described in the text.

[0042] S103, based on a predefined fusion loss function, performing hierarchical fusion on the semantically enhanced video features and the semantically enhanced text features to obtain multi-channel features of the video-text pair after feature fusion.

[0043] For example, layer-by-layer fusion can be performed through a CNN network based on a residual attention mechanism.

[0044] S104, performing text-video retrieval training based on a predefined text-video similarity function, a predefined multimodal retrieval loss function and multi-channel features of video-text pairs.

[0045] S105. Iteratively train based on the text-video retrieval results to optimize the fusion loss function, similarity function and multimodal retrieval loss function.

[0046] Specifically, based on the accuracy of the retrieval results, the coefficients and hyperparameters of the fusion loss function, the similarity coefficient of the text-video similarity function, and the loss coefficients and hyperparameters of the multimodal retrieval loss function can be optimized, so that the subsequent text-training video retrieval task based on the learned fusion loss function, similarity function and multimodal retrieval loss function can be more accurate.

[0047] The embodiment of the present invention provides a model training method for a cross-modal livestock behavior retrieval method. First, a training video and a description text matching the livestock behavior event of the training video are obtained, content semantic enhancement is performed on the training video and the description text, and time semantic enhancement is performed on the training video and the description text to obtain semantically enhanced video features and semantically enhanced text features; then, based on a predefined fusion loss function, the semantically enhanced video features and the semantically enhanced text features are layered fused to obtain feature fused video-text pair multi-channel features; then, based on a predefined text-video similarity function, a predefined multi-modal retrieval loss function and the video-text pair multi-channel features, text-video retrieval training is performed; finally, based on the text-video retrieval result, iterative training is performed to optimize the fusion loss function, the similarity function and the multi-modal retrieval loss function. First, by performing content semantic enhancement and temporal semantic enhancement on the training video, the essential semantic information of livestock behavior events in the video can be enriched. By performing content semantic enhancement and temporal semantic enhancement on the description text, the essential semantic information of the description text can be enriched, thereby obtaining the essential semantic information of the entire livestock behavior event under different modalities. Second, based on the video features describing livestock behavior events after semantic enhancement and the text features describing livestock behavior events after semantic enhancement, hierarchical features are fused into multi-channel features of video-text pairs. Through hierarchical feature fusion, the complementary features between the two different modalities of video and text can be combined, the redundant information of the fused features can be reduced, a more comprehensive understanding of the entire livestock behavior event can be obtained, and the semantic consistency between different modalities can be enhanced. It can effectively improve the accuracy of livestock video-text retrieval, reduce the cost of manual control, and realize accurate and intelligent real-time monitoring and management of a large number of livestock behaviors. For example, livestock behavior videos that users are concerned about can be accurately and quickly retrieved from massive agricultural livestock monitoring videos based on the input text entered by the user, and livestock with abnormal behaviors can be located.

[0048] Optionally, in the embodiment of the present invention, the content semantic enhancement process in S102 may include the following A01 to A02:

[0049] A01. Enhance the local features of training videos and the global features of training videos that describe livestock behavior events.

[0050] It can be understood that the local features in the training video can characterize the livestock behavior events from various presentation angles. The global features in the training video can characterize the livestock behavior events from the overall presentation angle.

[0051] For example, local features describing livestock behavior events in a training video include object features and motion features; global features describing livestock behavior events in a livestock training video include appearance features of the training video.

[0052] Among them, object features can represent livestock entities (such as cows) in livestock behavior events, motion features represent the movement of livestock entities in livestock behavior events (drinking water), and training appearance features can roughly describe the overall information of livestock behavior events (cows standing on the grass and drinking water).

[0053] A02. Enhance the local and global features of livestock behavior events in the description text.

[0054] It can be understood that the local features corresponding to the text indicate the features possessed by a specific part or segment in the text, and livestock behavior events can be characterized from multiple description angles. For example, the local features describing livestock behavior events in the text include nouns and verbs. The global features corresponding to the text describe various indicators and information of the overall features and attributes of the entire text, and can comprehensively describe livestock behavior events. For example, the global features describing livestock behavior events in the text include sentences. The sentences in the text can show the overall description of the event. In the embodiment of the present invention, sentences are used to represent the global semantic information of livestock behavior events, and nouns and verbs in the sentences are used to understand the entities and the behaviors of the entities in the sentences.

[0055] Based on this scheme, by semantically enhancing both the local features and the global features describing livestock behavior events in the training video, the semantic information of the local features and the global features of the livestock training video can be enhanced respectively, and by semantically enhancing both the local features and the global features describing livestock behavior events in the text, the semantic information of the local features and the global features of the text can be enhanced respectively, enriching the overall semantic information of the livestock behavior events and the semantic information of each detail, thereby enhancing the semantic consistency of the livestock training video and the description text, thereby improving the accuracy of subsequent retrieval based on the semantically enhanced features to locate the training video segments or images in the livestock training video where the livestock behaviors described in the description text are located.

[0056] Optionally, in the embodiment of the present invention, the training video content semantic enhancement process in A01 may include the following A11. Figure 2 As shown in , the content semantic enhancement process of the text in A02 may include the following A12 to A14:

[0057] A11. Extract object features, motion features, and appearance features of training videos.

[0058] It should be noted that the appearance feature can roughly describe the global information of the entire livestock behavior event in the training video, the cow drinking water, the object feature describes the local information of the training video of the livestock behavior event, the object is the cow, and the motion feature describes the local information of the training video of the livestock behavior event, the action is drinking water.

[0059] In the embodiment of the present invention, livestock behavior events are represented by extracting multi-channel features of training videos, that is, features of three training video channels, namely object features, motion features and appearance features, can be extracted from the training videos to characterize livestock behavior events from multiple angles.

[0060] For example, based on the object classification network, the object features of each frame sampled in the training video can be extracted to obtain a sequence of object features: Represents the deep feature vector of the oth frame in the i-th training video extracted by the object classification network. Where i is an integer from 1 to th, th represents the total number of training videos, and th is a positive integer.

[0061] For example, the behavior-based video retrieval network can extract the motion features of each frame sampled in the training video to obtain a sequence of motion features: represents the deep features of the oth frame in the ith training video extracted by the action video retrieval network.

[0062] For example, the training appearance features of each frame sampled in the th training video can be extracted by a pre-trained CNN (Convolutional Neural Networks) to obtain a sequence of appearance features: represents the deep features of the kth frame in the i-th training video extracted by the pre-trained CNN network.

[0063] For example, the sequence of appearance features is transformed into a NetVLAD (Network of Vectors of Locally Aggregated Descriptors, a deep learning model for image retrieval and visual localization) model. Aggregate into vector representation Get the global features of the training video; aggregate the object feature sequence of the training video into a vector representation through the NetVLAD model And the motion feature sequence of the training video Aggregated into a vector representation Get the local features of the training video.

[0064] It can be understood that by extracting semantic features describing livestock behavior events in training videos in a multi-channel form, compared with related technologies, more fine-grained features can be extracted from training videos, making livestock behavior events more distinguishable. That is, extracting surface features, object features, and motion features from training videos can more richly describe the essential semantics of livestock behavior events and improve the distinguishability of features. For example, rumination is a continuous action, including gastric flow and chewing, and generally 40s-50s of training video segments can be retrieved, while related technologies may only be able to find training videos of chewing for a few seconds, and the retrieval accuracy is low.

[0065] A12. Obtain the description sentences in the description text, and the nouns and verbs in the description sentences.

[0066] A13. Extract synonyms that describe nouns in a sentence and synonyms that describe verbs in a sentence.

[0067] For example, to describe the sentence "The cow is standing and chewing the rumination", we can extract the nouns "cow" and "grass", and the verbs "stand" and "chew".

[0068] A14. Embed synonyms of nouns and synonyms of verbs.

[0069] Generally, word embedding refers to mapping vocabulary to high-dimensional space vectors. Noun embedding is used to map nouns to vector space. Verb embedding is used to map verbs to vector space. Sentence embedding is used to convert sentences into fixed-length vectors.

[0070] It should be noted that when embedding, the extracted nouns, verbs, sentences, synonyms of nouns, and synonyms of verbs need to be converted into the form of embedding vectors before the embedding operation is performed.

[0071] Specifically, the embedding vector conversion operation can be performed through a pre-trained Word2vec model (a model for generating word embedding vectors).

[0072] Based on this scheme, on the one hand, the video appearance features (global features) in the training video can be extracted respectively, and the motion features and object features (two types of local features) in the training video can be extracted, that is, the video semantics representing livestock behavior events can be extracted from multiple channels; the description sentences (global features) in the description text, the verbs and nouns (local features) in the description sentences can be obtained respectively, and the text semantics representing livestock behavior events can be extracted from multiple channels. That is, the semantic information of livestock behavior events can be enriched from multiple channels first, so that the semantics of livestock behavior events to be retrieved can be accurately described from multiple aspects. On the other hand, for the description text, synonyms of nouns and synonyms of verbs can also be expanded, which can improve the accuracy and robustness of the retrieval in livestock videos based on the input text. Even if the vocabulary or sentence structure used by the user is different, the correct video or video frame of the livestock with the target livestock behavior event can be returned, which can improve the user experience and the practicality of the retrieval system.

[0073] Specifically, in the embodiment of the present invention, the above-mentioned A14 may specifically include the following steps A41 to A43:

[0074] A41. Aggregate the nouns and synonyms of the nouns in the description sentence to generate a noun set corresponding to the first noun search tag.

[0075] The first noun search tag is a predefined noun search tag, such as the actual search tags of the first noun and its synonyms.

[0076] It can be understood that embedding noun synonyms in each sentence containing nouns, that is, merging nouns and noun synonyms and adding them to the set of retrieval tags corresponding to the nouns, can enrich the noun semantic information of the descriptive text, such as enriching the description of livestock entities and enriching the location description of livestock behavior events.

[0077] For example, in the retrieval process, by searching for a synonym of noun 1 in the description text as noun 2, and noun 2 being a noun in a preset noun retrieval tag set, noun 2 is determined as a noun retrieval tag for retrieval.

[0078] A42. Aggregate the verbs and their synonyms in the description sentence to generate a verb set corresponding to the first verb search tag.

[0079] The first verb search tag is a predefined verb search tag, such as an actual search tag of a synonym of the first verb and the second gerund.

[0080] It can be understood that embedding verb synonyms in each sentence containing a verb, that is, merging the verb and its synonyms and adding them to the synonym set of the retrieval tag corresponding to the verb, can enrich the semantic information of the verb in the sentence, such as enriching the action description of the entity.

[0081] For example, based on the residual attention mechanism, a word (noun or verb) and its synonyms are merged, the weights of the word and its synonyms are calculated through the attention mechanism, the embedding of each synonym is multiplied by the corresponding weight and added to the set of words.

[0082] It can be understood that using the residual attention mechanism to fuse nouns and their synonyms, and to fuse verbs and their synonyms in text sentences can effectively enrich the semantic diversity of nouns and verbs, and enhance the core content semantic information of verbs and nouns in the text, that is, enrich the verb information and noun information describing livestock behavior events.

[0083] A43. Generate a first description sentence retrieval tag based on the first noun retrieval tag, the first verb retrieval tag and the structure of the extracted description sentence.

[0084] Among them, the first noun retrieval tag, the first verb retrieval tag and the first description sentence retrieval tag are matched with the livestock behavior events in the training video.

[0085] For example, sentence embedding vectors are extracted from description text using a pre-trained Word2vec model. The sentence embedding vector includes the embedding vector of each word in the sentence. The nouns and verbs are parsed from the sentence using the NLTK (Natural Language Toolkit) parser. The nouns parsed by the NLTK parser are converted into noun embedding vectors using the pre-trained Word2vec model. Convert verbs parsed by NLTK parser to verb embedding vectors Sentence embedding vectors, noun embedding vectors, and verb embedding vectors are embedded in NetVLAD Aggregate into a vector, and get the embedding vector of the multi-channel features of the extracted text as:

[0086] For example, the sentence embedding vector is represented as represents the set of sentence embedding vectors of the jth sentence in the description text matching the i-th training video, l se Represents the number of words in the jth sentence in the description text that matches the i-th training video.

[0087] The noun embedding vector is represented as represents the set of noun embedding vectors of the jth sentence in the description text matching the i-th training video, l no Represents the number of nouns in the jth sentence in the i-th training video.

[0088] The verb embedding vector is represented as represents the set of verb embedding vectors of the jth sentence in the description text matching the i-th training video, l ve represents the number of verbs in the jth sentence in the i-th training video.

[0089] In an embodiment of the present invention, for the description text matching the animal behavior event in the training video, the description sentence matching the livestock behavior event, the nouns in the description sentence and the verbs in the description sentence can be extracted first; then the extracted description sentence, nouns and verbs are converted into a description sentence embedding vector, as well as a noun embedding vector and a verb embedding vector; the description sentence embedding vector, the noun embedding vector and the verb embedding vector are respectively aggregated into one vector to obtain an embedding vector description of the extracted text feature.

[0090] Specifically, the synonyms of nouns and verbs are first extracted through the NLTK parser, and then the extracted synonyms of nouns and verbs are converted into embedded forms through the pre-trained Word2vec model. Then the synonyms of nouns and verbs are embedded.

[0091] For example, define the embedding vectors of noun synonyms in each sentence containing a noun as p represents the number of synonyms of the lth noun in the jth sentence of the description text matching the livestock behavior event in the i-th training video, and no1 represents the first synonym embedding vector of the lth noun.

[0092] For example, define the embedding vectors of verb synonyms in each sentence containing a verb as q represents the number of synonyms of the lth verb in the jth sentence of the description text matching the livestock behavior event in the i-th training video, and ve1 represents the first synonym embedding vector of the l-th verb.

[0093] In an embodiment of the present invention, a multi-channel feature embedding operation can be performed based on appearance features, object features, motion features, description sentences after content semantics enhancement, a noun set after synonym embedding, and a verb set after synonym embedding.

[0094] In the embodiment of the present invention, the multi-channel feature embedding process can be composed of a gated embedding function and a softmax function (activation function), and the gated embedding function can recalibrate the strength of different activations and enhance nonlinear interactions. For example, the embedding operation can be performed based on the following embedding method.

[0095]

[0096] in, Represents the video features extracted from the training video and the text features extracted from the description text, F∈{app,ob,mo,se,no,ve}. app represents the extracted appearance feature vector, ob represents the extracted object feature vector, mo represents the extracted motion feature vector, se represents the extracted description sentence vector, no represents the extracted noun and noun synonym vector, ve represents the extracted verb and verb synonym vector, are the coefficients learned in advance. Respectively and The dimension of the feature, BatchNorm means batch normalization, represents element-wise multiplication, such as Hadarmard (Hadamard product), σ represents element-wise activation function, such as sigmoid (activation function), and φ represents the softmax function.

[0097] when When only the features of the text are represented, The sub-tag i of can be ignored. When only the features of the training video are represented, The sub-tag j can be ignored. When representing the features of the training video and the features of the text, The sub-tag i of represents the i-th training video, and the sub-tag j represents the j-th sentence in the description text matching the i-th training video.

[0098] Based on this scheme, by embedding synonyms of verbs and nouns, the semantic information of the descriptive text can be enriched. The richer semantic information describing livestock behavior events, livestock entities and livestock entity movements can improve the accuracy of satisfying the retrieval conditions in the input-based descriptive text retrieval training video and improve the accuracy and robustness of the input-based input text retrieval in livestock videos. Even if the user uses different vocabulary or sentence structure, the correct livestock video or video frame with the target livestock behavior event can be returned, which can improve the user experience and the practicality of the retrieval system.

[0099] Optionally, in the embodiment of the present invention, the temporal semantics enhancement process in S102 may include the following B11 and B12:

[0100] B11. Based on the attention mechanism and the extracted surface features, the temporal semantic information describing livestock behavior events in the training videos is enhanced to obtain the surface features after temporal semantic enhancement.

[0101] For example, the attention mechanism can be used based on formula (1), the extracted appearance features are input into the CNN network to perform temporal semantic enhancement of the training video, and the parameters of the attention mechanism corresponding to the training video are output: and

[0102]

[0103] in, represents the hth appearance feature of the i-th training video, H represents the number of appearance features of the i-th training video, and W att is the learnable mapping parameter of the attention mechanism, The temporal semantic information of the h-th appearance feature, Represents the weight information of the hth appearance feature corresponding to the i-th training video output by the attention mechanism, It represents the temporal semantic information after the hth appearance feature of the i-th training video is enhanced.

[0104] B12. Based on the attention mechanism, GRU (Gated Recurrent Unit) and descriptive sentences, the temporal semantic information of the descriptive sentences of livestock behavior events in the description text is enhanced to obtain the descriptive sentences with enhanced temporal semantics.

[0105] For example, based on formula (2), the temporal semantic information describing the sentence vector can be extracted based on the attention mechanism, GRU and CNN network.

[0106]

[0107] in, represents the average embedding vector of the zth word in the jth sentence in the description text of the i-th training video input to the GRU, W gru represents the learnable mapping parameters of GRU, θ is the other learnable parameters of GRU, represents the embedded representation information of the jth sentence where the zth verb is output by GRU, The temporal semantic information of the zth verb in the jth sentence corresponding to the i-th training video output after temporal semantic enhancement; represents the weight information of the zth verb in the jth sentence corresponding to the i-th training video output by the attention mechanism, W represents the embedded representation information of the zth verb of the jth sentence corresponding to the i-th training video output by GRU, att is the learnable mapping parameter of the attention mechanism.

[0108] Based on this scheme, the time information describing livestock behavior in livestock videos can be enhanced based on the attention mechanism, and the time information describing livestock behavior in input text can be enhanced based on the attention mechanism and GRU. The semantic enhancement of cross-modal time can obtain the time essential semantic information of the entire event of livestock behavior, and can enhance the accuracy of livestock behavior retrieval. For example, when cattle normally graze, they lower their heads to gnaw and chew, and when cattle ruminate, they do not gnaw from the ground or from the feed trough, and chew the food stored in the stomach for a long time. By improving the time semantics of chewing, it is determined that the retrieved behavior is rumination, and it is determined that the duration of the chewing behavior should be greater than the preset duration. Behavior events with a chewing time less than the preset duration can be filtered out from the retrieved video, thereby improving the accuracy of cattle rumination behavior retrieval and reducing the probability of retrieving cattle's normal grazing behavior as cattle rumination behavior.

[0109] Alternatively, if Figure 2 As shown in , in the embodiment of the present invention, the layered fusion operation in S103 may specifically include the following C11 to C15:

[0110] C11. Fuse the extracted motion features and the extracted object features into motion-object pair features.

[0111] Among them, the fusion of motion features and object features of livestock training videos is the fusion of local features of training videos.

[0112] C12. The extracted motion features, motion-object pair features and temporal semantically enhanced appearance features are fused into video multi-channel features.

[0113] Among them, the fusion of motion features, motion-object pair features and temporal semantically enhanced appearance features is the fusion of global features of the training video.

[0114] C13. The verbs in the verb set and the nouns in the noun set corresponding to the description sentence are merged into word-noun pairs.

[0115] Among them, the words of the verb set and the noun set describing the sentence are fused into the fusion of local features of the text.

[0116] It should be noted that when fusing verb-noun pairs, the verbs and nouns are usually converted into vector forms for fusion to obtain verb-noun pairs in vector form.

[0117] C14. The extracted description sentences, verb-noun pairs, and description sentences enhanced with temporal semantics are fused into text multi-channel features.

[0118] Among them, the fusion of description sentences, verb-noun pairs and description sentences after time semantics enhancement is the fusion of global text features.

[0119] Similarly, the extracted description sentences and the description sentences after temporal semantic enhancement are converted into vector form and fused with the verb-noun pairs in vector form to obtain the semantically enhanced text multi-channel feature vector form.

[0120] C15. Fusion of video multi-channel features and text multi-channel features into training video-text pair features for livestock behavior event matching.

[0121] In other words, the hierarchical fusion model can first use the attention mechanism to fuse the local-level features of the training video and text to generate new local features, and then use the weighted average method to merge the new local features with the global-level features of the training video and the description text, so as to obtain the essential semantic information of the entire event of livestock behavior and enhance the semantic consistency between different modalities.

[0122] Exemplarily, in an embodiment of the present invention, local features and global features of the training video can be fused based on the residual attention mechanism, and local features and global features of the description text can be fused.

[0123] Among them, the residual attention mechanism can be described based on the following formula (3).

[0124]

[0125] Where u represents the mode of feature fusion, u∈{video,text}, that is, u indicates the training video mode or text mode; h represents the type of feature fusion, h∈{local,global}, that is, h indicates local feature fusion or global feature fusion; y u,h represents the result of feature fusion output, represents the g-th feature of the feature fusion input, and g represents the total number of features of the feature fusion input.

[0126] Based on the scheme, the present invention provides a hierarchical fusion model (HFM), hierarchical fusion is performed based on the HFM to fuse the multi-channel features of the semantically enhanced training video and the description text, the semantically enhanced local-level features and the global-level features are fused, the local features of the training video and the text can be fused respectively, and the fused local features are fused with the respective global features respectively, so that highly correlated complementary features in each modality are combined to extract the features of the global level (training video appearance and text sentences) to perceive the entire event of livestock behavior, while also paying attention to the local-level fine-grained features of specific entities and their movements (training video objects and text nouns, training video movements and text verbs) in the event, and mapping the features representing the same attributes to the same shared space, so as to obtain comprehensive semantic information of the entire event of livestock behavior, which can not only fully highlight the basic semantic information in the training video and the description text, but also reduce the redundant information of feature fusion.

[0127] Optionally, in the embodiment of the present invention, the above S104 may include the following D11-D14:

[0128] D11. Determine a first similarity between the video multi-channel feature and the text multi-channel feature.

[0129] D12. Determine the second similarity between the apparent features after temporal semantic enhancement and the description sentence after temporal semantic enhancement.

[0130] D13. Determine the third similarity of the motion-object pairs and the verb-noun pairs.

[0131] D14. Determine the video-text similarity based on the first similarity, the second similarity, the third similarity and a predefined similarity function.

[0132] The training video-text similarity function provided by the embodiment of the present invention includes the following formula (5).

[0133] S vt =βS fu +γS app-se +ηS moob-veno Formula (5)

[0134] Among them, S vt Indicates the similarity between video and text, S fu represents the first similarity (i.e., the similarity between the global features of the training video and the global features of the text after hierarchical fusion), S app-se represents the second similarity (i.e., the similarity between the global features of the training video and the global features of the description text before hierarchical fusion), S moob-venorepresents the third similarity (i.e., the similarity between the local features of the training video and the local features of the description text after hierarchical fusion), β represents the similarity fusion coefficient of the first similarity, γ represents the similarity fusion coefficient of the first similarity, and η represents the similarity fusion coefficient of the first similarity, β+γ+η=1.

[0135] It can be understood that the importance of the similarity of features at different angles is different. The importance of the feature similarity of each angle to the overall feature similarity can be learned in advance during the model training process. The importance of the feature similarity of each angle to the overall feature similarity can be adjusted by the weight coefficient corresponding to each feature angle similarity to obtain the overall feature similarity. The weight coefficient corresponding to each feature angle similarity is obtained. The initial values ​​of each similarity fusion coefficient can be the same, and the above-mentioned similarity fusion coefficients can be iteratively optimized based on the retrieval results. Optionally, in an embodiment of the present invention, the similarity coefficient between the text and the video is calculated to enhance the semantic consistency of different modalities, that is, to enhance the semantic consistency of the livestock video and the description text. Among them, multiple similarities of text and video can be aggregated by weight coefficients.

[0136] Based on this scheme, by training the similarities of the video and the description text from multiple angles, the final training video-text similarity is obtained comprehensively according to the similarity and the importance of the similarity (weight coefficient), which can improve the semantic consistency of different modalities and thus improve the accuracy of cross-modal retrieval.

[0137] Optionally, in the network training process of the text-training video retrieval model of the embodiment of the present invention, in the embodiment of the present invention, the fusion loss function of the training video-text pair is defined as the following formula (6). The fusion loss weight is learned by minimizing the fusion loss function. The fusion loss function uses a bi-directional maximum margin ranking loss to learn a feature embedding vector for training text-video cross-modal retrieval.

[0138]

[0139] Among them, S represents the feature similarity, S∈{S fu ,S app-se ,S moob-veno}, △ is the boundary constant, B represents the number of samples (batch size), δ S (v i ,t b ) represents the similarity score between the feature of the i-th training video and the b-th text label, δ S (v i ,t i ) represents the features and text tags of the i-th training video t i The similarity score of S (v b,t i ) represents the training video v b The features and text labels t i The similarity score of , the text tags include noun retrieval tags and verb retrieval tags.

[0140] Specifically, positive and negative correlations can indicate whether the text and training video are synchronized. i and mark t i Same as above, or livestock training video v i and text marker t b If both are negative, choose a smaller sample size. Otherwise, choose a larger sample size.

[0141] Optionally, during the network training process, the retrieval loss weights of each feature can be learned by minimizing the multimodal retrieval loss function and the reference multi-task loss fusion based on the multimodal retrieval loss function shown in the following formula (6).

[0142] L = LossFusion(λL fu +μL app-se +σL moob-veno ) Formula (7)

[0143] Among them, L fu represents the first retrieval loss of the video multi-channel feature and the text multi-channel feature, L app-se represents the apparent features after temporal semantic enhancement and the second retrieval loss of the description sentence after temporal semantic enhancement, L moob-veno represents the third retrieval loss of the motion-object pair, λ represents the loss weight of the first retrieval loss, μ represents the loss weight of the second retrieval loss, and σ represents the loss weight of the third retrieval loss.

[0144] It can be understood that through iterative operations, when the retrieval accuracy is stable, the learning is stopped, the video-text similarity function is determined based on the learned generated similarity fusion coefficient and hyperparameters, the fusion loss function is determined based on the learned fusion loss weight and hyperparameters, and the multimodal retrieval loss function is determined based on the learned retrieval loss weight and hyperparameters. And the matching relationship between the video multi-channel features and the text multi-channel features of the livestock behavior events, as well as the retrieval labels of the livestock behavior events, the synonym sets of the noun retrieval labels, the synonym sets of the verb retrieval labels, and the set of sentence retrieval labels are saved.

[0145] Figure 3 A semantically enhanced logic diagram provided by an embodiment of the present invention, such as Figure 3As shown in the figure, the event is a cow ruminating. Enhance the temporal semantic information describing the event in the video. Enhance from video frame 1-1 to video frame 1-2 to video frame 2-1 to video frame 2-2. Video frame 2-1 is before video frame 1-1, and video frame 2-2 is after video frame 1-2. Enhance the content semantic information of verbs and nouns in the text and the temporal semantic information of sentences. Is the cow or the calf ruminating? Synonyms of cow include "livestock, native livestock, black bull, Shen cattle, yellow cattle, yak, beef cattle, dairy cow...", synonyms of calf include "calf, yellow calf, calf, calf, young cow...", synonyms of rumination include "chewing, chewing, chewing grass, foaming...", "Is the cow or the calf ruminating?", synonyms of cow include "cattle, bull, buffalo, ox, steer", synonyms of calf include "little-cow, heifer, bullock", synonyms of ruminating include "chew the cud, regurgitate, remasticate", and temporal semantic enhancement is performed through the tenses of "is" and "ruminating". The semantic consistency between the texts of the video is improved through the multi-channel feature representation enhancement model (i.e. the above-mentioned content semantic enhancement processing, temporal semantic enhancement processing and layered fusion processing).

[0146] Figure 4A schematic diagram of a cross-modal livestock behavior retrieval training model provided by an embodiment of the present invention, wherein the cross-modal livestock behavior retrieval training model 400 includes: a data input module 401, a semantic enhancement module 402, a hierarchical feature fusion module 403 and a retrieval training module 404; the semantic enhancement module 402 includes: a video content semantic enhancement module 4001, a text content semantic enhancement module 4002 and a time semantic enhancement module 4003. The video content semantic enhancement module 4001 includes: a video feature extraction module 4010. The text content semantic enhancement module 4002 includes: a text feature extraction module 4020 and a feature embedding module 4030. The video feature extraction module 4010 includes: an object feature extraction module 4011, a motion feature extraction module 4012, and an appearance feature extraction module 4013. The text feature extraction module 4020 includes: a sentence feature extraction module 4021 and a word feature extraction module 4022. The hierarchical feature fusion module 403 includes: a video multi-channel feature fusion module 4031, a text multi-channel feature fusion module 4032 and a video-text multi-channel feature fusion module 4033. The retrieval training module 404 includes: a similarity determination module 4041 and a loss determination module 4042. The similarity determination module 4041 includes the video-text similarity function in the above formula (5), and the loss determination module 4042 includes the fusion loss function of the training video-text pair in the above formula (6) and the multimodal retrieval loss function in the above formula (7).

[0147] It should be noted that, in an embodiment of the present invention, a cross-modal livestock behavior retrieval training logic diagram is also provided, such as Figure 5 As shown in Figure 5A cross-modal livestock behavior retrieval training logic diagram provided in an embodiment of the present invention. For the livestock video training process, a multi-stream feature representation enhancement model (Multi-stream Features Representation Enhancement, MFRE) is provided to enhance the content semantic information and temporal semantic information describing livestock behavior events in the video, and perform multi-channel feature extraction of video appearance, motion and objects. For the input text training process, the multi-channel feature representation enhancement model is used to enhance the content semantic information and temporal semantic information describing livestock behavior events in the text, and perform multi-channel feature extraction and embedding of text (sentence) nouns, verbs and synonyms of related nouns and verbs. Specifically, the pre-trained CNN network (i.e., appearance feature extraction model), object classification network (i.e., object feature extraction model) and pre-trained behavior retrieval network (i.e., action feature extraction model) are used to extract the appearance, motion and object features in the video, and then NetVLAD is used to aggregate them into three single vector representations corresponding to appearance, motion and object; at the same time, the pre-trained Word2vec model and NLTK parser are used to extract the features of sentences, nouns, verbs and corresponding noun and verb synonyms in the text, and then NetVLAD is used to aggregate them into three single vector representations corresponding to sentences, nouns and verbs. The diversity of semantic relevance and hierarchical fusion model are used to aggregate features between different modalities, that is, the hierarchical fusion of local features and global features of video and text multimodalities is performed, and video-text cross-modal semantic enhancement is performed to enhance the semantic consistency between video and text in the livestock cross-modal behavior retrieval task, reduce the redundant information of fused features, and obtain a more comprehensive understanding of the entire event.

[0148] Model application process:

[0149] The following is an application process of a cross-modal livestock behavior retrieval method provided by an embodiment of the present invention.

[0150] Figure 6 A schematic flow chart of a cross-modal livestock behavior retrieval method provided by an embodiment of the present invention is shown in FIG. Figure 6 As shown in , the following S601 to S606 may be included:

[0151] S601: Acquire livestock videos and input text input by a user.

[0152] Wherein, the input text includes livestock behavior events.

[0153] For example, the input text may be text inputted from livestock behavior retrieved according to the needs of a user (eg, a breeder, a veterinarian, or a researcher, etc.).

[0154] It can be understood that due to differences among different users, the descriptions of the input texts input by the users are diverse and different.

[0155] S602: Extract the target sentence, nouns and verbs of the target sentence from the input text.

[0156] S603: Determine the retrieval tags of nouns and verbs of the target sentence based on the word set corresponding to the livestock behavior retrieval tags.

[0157] The word set corresponding to the livestock behavior search tag includes: a synonym set of the search tag of nouns and a synonym set of the search tag of verbs.

[0158] S604: Searching for video features that match the noun retrieval tag, the verb retrieval tag, and the sentence structure of the target sentence from the video-text multi-channel feature set.

[0159] Among them, the video-text multi-channel feature set includes fusion features after temporal semantic enhancement and content semantic enhancement of the descriptive text and video corresponding to the livestock behavior events.

[0160] S605: Retrieve livestock behavior events in the livestock video based on the matched video features.

[0161] Optionally, the search result of the livestock of the livestock behavior event may be a video of a specific livestock having the livestock behavior event, or may be a video frame, which may be set according to user requirements and is not specifically limited in the embodiment of the present invention.

[0162] The embodiment of the present invention provides a livestock behavior cross-modal retrieval method. In the first aspect, a livestock video and an input text input by a user are obtained, and based on the word set corresponding to the livestock behavior retrieval tag, the retrieval tag of the noun of the target sentence and the retrieval tag of the verb of the target sentence are determined, and then from the video-text multi-channel feature set, video features that match the noun retrieval tag, the verb retrieval tag and the sentence structure of the target sentence are searched, and finally, based on the matching video features, livestock behavior events in the livestock video are retrieved. In the first aspect, since the word set corresponding to the livestock behavior retrieval tag includes: a synonym set of the noun retrieval tag and a synonym set of the verb retrieval tag. That is, by inputting the synonyms of nouns and verbs in the text, the noun retrieval tags and verb retrieval tags used for retrieval can be found. The essential semantic information of the livestock behavior event to be retrieved can be accurately described by the noun retrieval tags, verb retrieval tags and the sentence structure of the target sentence. Then, the video features matching the text features are found in the video-text multi-channel feature set. Since the video-text multi-channel feature set includes the fusion features after the description text and video corresponding to the livestock behavior event are enhanced in time semantics and content semantics. The fusion features have a more comprehensive understanding of the entire livestock behavior event, and the semantic consistency between different modalities is stronger. Therefore, the video features found can more accurately represent the livestock behavior events to be retrieved in the description text. Retrieving livestock videos based on the matching video features can effectively improve the accuracy of livestock video-text retrieval, reduce the cost of human control, and realize accurate and intelligent real-time monitoring and management of a large number of livestock behaviors. For example, livestock behavior videos that users are concerned about can be accurately and quickly retrieved from a large number of agricultural livestock monitoring videos according to the input text entered by the user, and livestock with abnormal behaviors can be located.

[0163] Optionally, in the livestock behavior cross-modal retrieval method provided in the embodiment of the present invention, the above S603 may include the following E1:

[0164] E1. Based on the pre-trained similarity function and the pre-trained retrieval loss function, find the object features, motion features and appearance features that match the retrieval tag of the noun, the retrieval tag of the verb and the target sentence structure.

[0165] The motion feature represents the motion of the livestock entity in the livestock behavior event in the video, the object feature represents the livestock entity in the livestock behavior event in the livestock video, and the appearance feature describes the livestock behavior event in the video.

[0166] For example, the video-text similarity function pre-trained based on the above formula (5) can be used to calculate the video-text similarity between the livestock video and the input text. The multimodal retrieval loss function pre-trained based on the above formula (7) can be used to perform video-text retrieval and output the retrieval results.

[0167] Based on this scheme, the matching object features, motion features and appearance features can be accurately found through the above-mentioned trained similarity function and pre-trained retrieval loss function.

[0168] Corresponding to the above-mentioned method embodiments, the present invention also provides embodiments of a device and a terminal to which the device is applied.

[0169] The embodiments of the cross-modal livestock behavior retrieval method of the present invention can be applied to a computer device, such as a server or a terminal device. The method embodiments can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, the processor of the cross-modal livestock behavior retrieval device reads the corresponding computer program instructions in the non-volatile memory into the memory and runs them. From the hardware level, if Figure 7 As shown in FIG. 1 , a hardware structure diagram of a computer device in which a cross-modal livestock behavior retrieval system provided in an embodiment of the present invention is located is shown in FIG. Figure 7 In addition to the processor 710, memory 730, network interface 720, and non-volatile memory 740 shown, the server or electronic device where the device 731 is located in the embodiment may also include other hardware according to the actual function of the computer device, which will not be described in detail.

[0170] Figure 8 A schematic diagram of a cross-modal livestock behavior retrieval device is provided for an embodiment of the present invention. Figure 8As shown in (a) of FIG. 1 , the cross-modal livestock behavior retrieval device 800 includes: a data acquisition module 801, a feature extraction module 802, a retrieval tag determination module 803, a feature matching module 804 and a retrieval module 805; the data acquisition module 801 is used to acquire livestock videos and input text input by users, and the input text includes livestock behavior events; the feature extraction module 802 is used to extract the nouns and verbs of the target sentence and the target sentence in the input text; the retrieval tag determination module 803 is used to determine the retrieval tags of the nouns and the verbs of the target sentence based on the word set corresponding to the livestock behavior retrieval tag Retrieval tag; feature matching module 804, used to find video features that match the noun retrieval tag, the verb retrieval tag and the sentence structure of the target sentence from the video-text multi-channel feature set; retrieval module 805, used to retrieve livestock behavior events in livestock videos based on the matched video features; wherein the word set corresponding to the livestock behavior retrieval tag includes: a synonym set of the noun retrieval tag, and a synonym set of the verb retrieval tag; the video-text multi-channel feature is a fusion feature after the description text and video corresponding to the livestock behavior event are both enhanced in time semantics and content semantics.

[0171] Optionally, the feature matching module 804 is specifically used to find object features, motion features and appearance features that match the retrieval tags of nouns, the retrieval tags of verbs and the target sentence structure based on a pre-trained similarity function and a pre-trained retrieval loss function; wherein the motion features represent the movement of livestock entities in livestock behavior events in the video, the object features represent livestock entities in livestock behavior events in the livestock video, and the appearance features describe livestock behavior events in the video.

[0172] Optionally, combined Figure 8 (a) in Figure 8As shown in (b) of the figure, the cross-modal livestock behavior retrieval device 800 also includes: a time semantic enhancement module 806, a content semantic enhancement module 807, a hierarchical feature fusion module 808 and a retrieval training module 809; before acquiring the livestock video and the input text input by the user, the data acquisition module 801 is also used to acquire a training video and a description text matching the livestock behavior event in the training video, where the training video is any video in the training video set; the time semantic enhancement module 806 is used to perform content semantic enhancement on the training video and the description text, and the content semantic enhancement module 807 is used to perform time semantic enhancement on the training video and the description text. The method further comprises the following steps: a first step of performing a text-to-video retrieval training module 809 and a second step of iteratively training the text-to-video retrieval results. The first step of iteratively training the text-to-video retrieval results. The second step of iteratively training the text-to-video retrieval results. The first step of iteratively training the text-to-video retrieval results. The second step of iteratively training the text-to-video retrieval results. The third step of iteratively training the text-to-video retrieval results. The first ...

[0173] Optionally, the content semantic enhancement module 807 is specifically used to: enhance the video local features and video global features describing livestock behavior events in the training video, and enhance the text local features and text global features describing livestock behavior events in the description text.

[0174] Optionally, the content semantic enhancement module 807 is specifically used to: extract object features, motion features and appearance features of the training video; obtain descriptive sentences in the descriptive text, nouns and verbs in the descriptive sentences, extract synonyms of nouns in the descriptive sentences and synonyms of verbs in the descriptive sentences, and embed synonyms of nouns and synonyms of verbs.

[0175] Optionally, the content semantic enhancement module 807 is specifically used to: aggregate nouns and synonyms of nouns in the description sentence to generate a noun set corresponding to the first noun retrieval tag; aggregate verbs and synonyms of verbs in the description sentence to generate a verb set corresponding to the first verb retrieval tag; generate a first description sentence retrieval tag based on the first noun retrieval tag, the first verb retrieval tag and the extracted structure of the description sentence; wherein the first noun retrieval tag, the first verb retrieval tag and the first description sentence retrieval tag match the livestock behavior events in the training video.

[0176] Optionally, the temporal semantic enhancement module 806 is specifically used to: enhance the temporal information describing livestock behavior events in the training video based on the attention mechanism and the extracted surface features to obtain the surface features after temporal semantic enhancement; enhance the temporal semantic information of the description sentences describing livestock behavior events in the description text based on the attention mechanism, the gated recurrent unit Gated Recurrent Unit (GRU) and the description sentences to obtain the description sentences after temporal semantic enhancement.

[0177] Optionally, the hierarchical feature fusion module 808 is specifically used to: fuse the extracted motion features and the extracted object features into motion-object pair features; fuse the extracted motion features, motion-object features and the apparent features after time semantics enhancement into video multi-channel features; based on the structure of the description sentence, fuse the verbs in the verb set and the nouns in the noun set corresponding to the description sentence into verb-noun pairs; fuse the extracted description sentences, verb-noun pairs and the description sentences after time semantics enhancement into text multi-channel features; fuse the video multi-channel features and the text multi-channel features into video-text pair multi-channel features for matching livestock behavior events.

[0178] Optionally, the retrieval training module 809 is also used to: determine a first similarity between video multi-channel features and text multi-channel features; determine a second similarity between the apparent features after temporal semantic enhancement and the description sentences after temporal semantic enhancement; determine a third similarity between motion-object pairs and verb-noun pairs; and determine the video-text similarity based on the first similarity, the second similarity, the third similarity and a predefined similarity function.

[0179] Optionally, the predefined similarity function includes: S vt =βS fu +γS app-se +ηS moob-veno ; Among them, S vt Indicates the similarity between video and text, S fu Indicates the first similarity, S app-se Represents the second similarity, S moob-veno represents the third similarity, β represents the similarity fusion coefficient of the first similarity, γ represents the similarity fusion coefficient of the second similarity, η represents the similarity fusion coefficient of the third similarity, β+γ+η=1.

[0180] Optionally, the multimodal retrieval loss function includes: L = LossFusion (λL fu +μL app-se +σL moob-veno ), where L fu represents the first retrieval loss of video multi-channel features and text multi-channel features, L app-serepresents the second retrieval loss of the temporal semantically enhanced appearance features and the temporal semantically enhanced description sentence, L moob-veno represents the third retrieval loss of the motion-object pair, λ represents the loss weight of the first retrieval loss, μ represents the loss weight of the second retrieval loss, and σ represents the loss weight of the third retrieval loss.

[0181] Optionally, the fusion loss function includes: Among them, L S represents feature fusion loss, S represents feature similarity, S∈{S fu ,S app-se ,S moob-veno}, △ is the boundary constant, B represents the number of training samples, δ S (v i ,t b ) represents the similarity score between the i-th video and the b-th text label, δ S (v i ,t i ) represents the similarity score between the i-th video and the i-th text label, δ S (v b ,t i ) represents the similarity score between the bth video and the i-th text tag, and the text retrieval tags include the retrieval tags of nouns and the retrieval tags of verbs.

[0182] The cross-modal livestock behavior retrieval device provided by the embodiment of the present invention, in the first aspect, can enrich the essential semantic information of livestock videos by performing content semantic enhancement and time semantic enhancement on livestock videos, and can enrich the essential semantic information of input texts by performing content semantic enhancement and time semantic enhancement on livestock input texts, thereby obtaining the essential semantic information of the entire event under different modalities; in the second aspect, based on the features of livestock videos after semantic enhancement and the features of input texts after semantic enhancement, hierarchical fusion is performed into video multi-channel features and text multi-channel features. Through hierarchical fusion, the complementary features between the two different modalities of video and text can be combined, the redundant information of the fused features can be reduced, a more comprehensive understanding of the entire livestock behavior event can be obtained, and the semantic consistency between different modalities can be enhanced. The accuracy of livestock video-text retrieval can be effectively improved, the cost of human control can be reduced, and precise and intelligent real-time monitoring and management of a large number of livestock behaviors can be realized. For example, livestock behavior videos that users are concerned about can be accurately and quickly retrieved from massive agricultural livestock monitoring videos according to the input text entered by the user, and livestock with abnormal behaviors can be located.

[0183] Correspondingly, the present invention also provides a cross-modal livestock behavior retrieval device, which includes a processor; a memory for storing processor executable instructions; wherein the processor is configured to: execute the steps of the cross-modal livestock behavior retrieval method executed by the above-mentioned controller.

[0184] In one embodiment, a computer device is provided, comprising: a memory and a processor, wherein the memory stores a computer program, and the processor implements any step in the above cross-modal livestock behavior retrieval method when executing the computer program.

[0185] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, any step in the above cross-modal livestock behavior retrieval method can be implemented.

[0186] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, which will not be repeated here.

[0187] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Those of ordinary skill in the art can understand and implement it without paying creative labor.

[0188] The above describes specific embodiments of the present invention. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0189] Those skilled in the art will readily appreciate other embodiments of the invention after considering the specification and practicing the invention claimed herein. The invention is intended to cover any variations, uses or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not claimed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the following claims.

[0190] It should be understood that the present invention is not limited to the exact construction that has been described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

[0191] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A cross-modal livestock behavior retrieval method, characterized in that: The method comprises: Acquire a livestock video and input text input by a user, wherein the input text includes livestock behavior events; Extracting a target sentence from the input text, and nouns and verbs of the target sentence; Determining the retrieval tags of nouns and verbs of the target sentence based on the word set corresponding to the livestock behavior retrieval tags; Searching for video features that match the noun retrieval tag, the verb retrieval tag, and the sentence structure of the target sentence from a video-text multi-channel feature set; Retrieving livestock behavior events in the livestock video based on the matched video features; Among them, the word set corresponding to the livestock behavior retrieval tag includes: a synonym set of the noun retrieval tag, and a synonym set of the verb retrieval tag; the video-text multi-channel feature set includes a fusion feature after the description text and video corresponding to the livestock behavior event are enhanced in time semantics and content semantics.

2. The method according to claim 1, characterized in that The step of searching the video features matching the noun label, the verb label and the target sentence structure from the video-text multi-channel feature set includes: Based on the pre-trained similarity function and the pre-trained retrieval loss function, searching for object features, motion features, and appearance features that match the retrieval tag of the noun, the retrieval tag of the verb, and the target sentence structure; The motion feature represents the motion of the livestock entity in the livestock behavior event in the video, the object feature represents the livestock entity in the livestock behavior event in the livestock video, and the appearance feature describes the livestock behavior event in the video.

3. The method according to claim 1 or 2, characterized in that: Before acquiring the livestock video and the input text input by the user, the method further includes: Obtaining a training video and a description text matching a livestock behavior event in the training video, wherein the training video is any one of the training video sets; Performing content semantic enhancement on the training video and the description text, and performing time semantic enhancement on the training video and the description text to obtain semantically enhanced video features and semantically enhanced text features; Based on a predefined fusion loss function, performing hierarchical feature fusion on the semantically enhanced video features and the semantically enhanced text features to obtain a multi-channel feature of the video-text pair after feature fusion; Performing text-video retrieval training based on a predefined text-video similarity function, a predefined multimodal retrieval loss function, and the multi-channel features of the video-text pair; Based on the iterative training of the text-video retrieval results, the fusion loss function, the similarity function and the multimodal retrieval loss function are optimized.

4. The method according to claim 3, characterized in that: The content semantic enhancement of the training video and the description text includes: The local video features and the global video features describing the livestock behavior events in the training video are enhanced, and the local text features and the global text features describing the livestock behavior events in the description text are enhanced.

5. The method according to claim 4, characterized in that The method of enhancing the local video features and the global video features of the livestock behavior events in the training video, and enhancing the local text features and the global text features of the livestock behavior events in the description text, includes: Extracting object features, motion features, and appearance features of the training video; The description sentences in the description text, the nouns and verbs in the description sentences are obtained, the synonyms of the nouns in the description sentences and the synonyms of the verbs in the description sentences are extracted, and the synonyms of the nouns and the synonyms of the verbs are embedded.

6. The method according to claim 5, characterized in that The synonyms embedded in the noun and the synonyms embedded in the verb include: Aggregating the nouns and synonyms of the nouns in the description sentence to generate a noun set corresponding to the first noun search tag; Aggregating the verbs in the description sentence and the synonyms of the verbs to generate a verb set corresponding to the first verb search tag; generating a first description sentence retrieval tag based on the first noun retrieval tag, the first verb retrieval tag and the extracted structure of the description sentence; Among them, the first noun retrieval tag, the first verb retrieval tag and the first descriptive sentence retrieval tag match the livestock behavior events in the training video.

7. The method according to claim 6, characterized in that The performing temporal semantic enhancement on the training video and the description text corresponding to the training video includes: Based on the attention mechanism and the extracted surface features, the temporal information describing the livestock behavior events in the training video is enhanced to obtain surface features with temporal semantic enhancement; Based on the attention mechanism, the gated recurrent unit and the description sentence, the temporal semantic information of the description sentence describing the livestock behavior event in the description text is enhanced to obtain the description sentence with enhanced temporal semantics.

8. The method according to claim 7, characterized in that The step of performing hierarchical feature fusion on the semantically enhanced video features and the semantically enhanced text features to obtain a multi-channel feature of a video-text pair after feature fusion includes: fusing the extracted motion features and the extracted object features into motion-object pair features; The extracted motion features, the motion-object features and the temporal semantically enhanced appearance features are fused into a video multi-channel feature; Based on the structure of the description sentence, the verbs in the verb set and the nouns in the noun set corresponding to the description sentence are merged into a verb-noun pair; The extracted description sentences, the verb-noun pairs and the description sentences enhanced with time semantics are fused into text multi-channel features; The video multi-channel features and text multi-channel features are integrated into video-text pair multi-channel features for livestock behavior event matching.

9. The method according to claim 8, characterized in that The method of performing text-video retrieval training based on a predefined text-video similarity function, a predefined multimodal retrieval loss function and the multi-channel features of the video-text pair further includes: Determining a first similarity between the video multi-channel feature and the text multi-channel feature; Determine a second similarity between the temporal semantically enhanced apparent feature and the temporal semantically enhanced description sentence; determining a third similarity between the motion-object pair and the verb-noun pair; Based on the first similarity, the second similarity, the third similarity and the predefined similarity function, a video-text similarity is determined.

10. The method according to claim 9, characterized in that The predefined similarity function includes: vt =βS fu +γS app-se +ηS moob-veno ; Among them, S vt Indicates the similarity between video and text, S fu represents the first similarity, S app-se represents the second similarity, S moob-veno represents the third similarity, β represents the similarity fusion coefficient of the first similarity, γ represents the similarity fusion coefficient of the second similarity, η represents the similarity fusion coefficient of the third similarity, β+γ+η=1.

11. The method according to claim 10, characterized in that The multimodal retrieval loss function includes: L = LossFusion (λL fu +μL app-se +σL moob-veno ); Among them, L fu represents the first retrieval loss of the video multi-channel feature and the text multi-channel feature, L app-se represents the apparent features after temporal semantic enhancement and the second retrieval loss of the description sentence after temporal semantic enhancement, L moob -veno represents the third retrieval loss of the motion-object pair, λ represents the loss weight of the first retrieval loss, μ represents the loss weight of the second retrieval loss, and σ represents the loss weight of the third retrieval loss.

12. The method according to claim 11, characterized in that The fusion loss function includes: Among them, L S represents feature fusion loss, S represents feature similarity, S∈{S fu ,S app-se ,S moob-veno }, △ is the boundary constant, B represents the number of training samples, δ S (v i ,t b ) represents the similarity score between the i-th video and the b-th text label, δ S (v i ,t i ) represents the similarity score between the i-th video and the i-th text label, δ S (v b ,t i ) represents the similarity score between the bth video and the i-th text tag, and the text retrieval tag includes a noun retrieval tag and a verb retrieval tag.

13. A cross-modal livestock behavior retrieval device, characterized in that: The cross-modal livestock behavior retrieval device comprises: a data acquisition module, a feature extraction module, a retrieval tag determination module, a feature matching module and a retrieval module; The data acquisition module is used to acquire livestock videos and input text input by a user, wherein the input text includes target livestock behavior events; The feature extraction module is used to extract the target sentence, the nouns of the target sentence and the verbs of the target sentence from the input text; The retrieval tag determination module is used to determine the retrieval tags of nouns and verbs of the target sentence based on the word set corresponding to the livestock behavior retrieval tags; The feature matching module is used to search for video features that match the noun retrieval tag, the verb retrieval tag and the sentence structure of the target sentence from the video-text multi-channel feature set; The retrieval module is used to retrieve livestock behavior events in the livestock video based on the matched video features; wherein the word set corresponding to the livestock behavior retrieval tag includes: a synonym set of the retrieval tag of nouns and a synonym set of the retrieval tag of verbs; the video-text multi-channel feature is a fusion feature after the description text and video corresponding to the livestock behavior event are both enhanced in time semantics and content semantics.

Images