Livestock and poultry breeding whole-process quality information management method based on multi-modal AI

Abstract

This invention relates to the field of artificial intelligence technology in smart animal husbandry, and discloses a method for managing the quality information of the entire livestock and poultry breeding process based on multimodal AI. The method includes parsing unstructured records such as video streams and natural language descriptions, automatically fusing them to generate a dynamic event library containing events, time, location, and individual identifiers. Based on the event library, breeding stages are divided and stage features are extracted to form a breeding stage sequence. Spatial relationships between locations are analyzed to construct a location topology map of the breeding environment. Combining the location topology map with the breeding stage sequence, a set of subsequent operational instructions to be executed for individual livestock or poultry or groups is derived. This method solves the problems of fragmented multi-source information and unclear spatial relationships in the breeding process, realizing a closed loop from perception to decision-making, and improving the automation and accuracy of management.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology in smart farming, specifically to a method for managing the quality information of the entire livestock and poultry farming process based on multimodal AI. Background Technology

[0002] Quality management in livestock and poultry farming relies heavily on video surveillance and manual recording. While video streams are continuously generated, they lack semantic annotations, making it difficult to directly link them to specific business events. Manual recordings primarily describe key operations in natural language, forming another information stream. Existing technologies typically store and process these two elements independently, resulting in a disconnect between visual information and event semantics. Core elements such as events, subjects, time, and location are scattered across different data sources, failing to automatically integrate and form a complete, traceable sequence of farming activities.

[0003] In the spatial management of aquaculture environments, existing methods mostly use identifiers to register physical locations, such as enclosure numbers. This approach treats locations merely as static labels and fails to represent the inherent logical and physical relationships between different locations. Consequently, management systems struggle to understand the spatial topology of the environment and cannot make automated scheduling decisions based on location relationships.

[0004] A method is needed to correlate and parse parallel video and text records to automatically construct structured sequences of aquaculture events. Deep modeling of location information within the aquaculture environment is required, along with parsing and constructing its topological relationship network. Furthermore, integrating dynamic event sequences with spatial topology models is necessary to provide a foundation for precise planning and automated generation of aquaculture operations. Summary of the Invention

[0005] The purpose of this invention is to provide a method for quality information management of the entire livestock and poultry breeding process based on multimodal AI, so as to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, this invention provides a method for managing the quality information of the entire livestock and poultry breeding process based on multimodal AI, the method comprising:

[0007] The collected unstructured records of livestock and poultry farming are analyzed and transformed. The unstructured records include video stream data and natural language description data. By parsing the video stream data and natural language description data, a dynamic event database containing farming events, time, location and identification of participating livestock and poultry is created.

[0008] Based on the dynamic event library, the continuous timeline is divided into multiple breeding stages according to the natural division of the breeding process, and a feature representation that can represent the core quality status information of each breeding stage is extracted to form a breeding stage sequence.

[0009] All physical or logical locations mentioned in the unstructured records are parsed to determine the inclusion, adjacency, or path reachability relationships between locations, and a location topology map describing the spatial connectivity within the aquaculture environment is constructed.

[0010] By combining the location topology map with the breeding stage sequence, a set of breeding operation instructions that should be executed in the subsequent breeding process can be derived for a specific livestock individual or group.

[0011] Preferably, the collected unstructured records of livestock and poultry farming are analyzed and transformed. These unstructured records include video stream data and natural language description data. By parsing the video stream data and natural language description data, a dynamic event database is created, containing farming events, times, locations, and identifiers of participating livestock and poultry.

[0012] The video stream data is scanned frame by frame to identify individual livestock, poultry, feeders, equipment, and specific behaviors in the scene. The identification results are associated with the corresponding timestamps to generate visual feature segments with time stamps.

[0013] Natural language description data is parsed to extract event names, times of occurrence, locations involved, and individual or group numbers of livestock and poultry that describe breeding activities, forming structured text records;

[0014] The time-stamped visual feature fragments are spatiotemporally aligned with the structured text records. When the events identified in the visual feature fragments are consistent with the events described in the text records in terms of time and location, the two information are merged.

[0015] The merged event information is arranged in chronological order of occurrence and stored in a uniform entry format. Each entry contains at least the event type, start time, end time, associated location, and livestock identifier, thereby creating the dynamic event library.

[0016] Preferably, based on the dynamic event library, the continuous timeline is divided into multiple breeding stages according to the natural division of the breeding process, and a feature representation that can represent the core quality status information of each breeding stage is extracted to form a breeding stage sequence, including:

[0017] Traverse all event entries in the dynamic event library, and define a continuous occurrence interval of a specific type of event as an independent breeding stage based on the preset breeding process knowledge;

[0018] For each independent breeding stage, from all event entries belonging to the breeding stage, the core operation types frequently performed within the breeding stage, the material information used, and the equipment status involved are summarized as a stage behavior summary.

[0019] Extract the state change parameters of all relevant livestock and poultry individuals during the breeding stage from the dynamic event database. The state change parameters include changes in activity level, changes in feeding frequency, and weight gain. Then, aggregate and calculate these parameters.

[0020] The stage behavior summary and the aggregated state change parameters are combined and encoded into a fixed-dimensional numerical vector, which is the feature representation of the core quality state information of the breeding stage.

[0021] Arrange all breeding stages in chronological order and connect their corresponding feature representations sequentially to form the breeding stage sequence.

[0022] Preferably, all physical or logical locations mentioned in the unstructured records are parsed to determine the inclusion, adjacency, or path reachability relationships between the locations, constructing a location topology map describing the spatial connectivity within the aquaculture environment, including:

[0023] Extract a unique set of location names from all event entries in the dynamic event library, and standardize each location name.

[0024] By analyzing the contextual information about location descriptions in natural language description data and the movement trajectories of livestock or people in video stream data, it can be inferred whether there is a direct physical connection or logical jurisdiction relationship between any two locations.

[0025] Each standardized location name is abstracted into a node. If it is determined that there is a direct connection or jurisdiction relationship between two locations, an edge is established between the corresponding two nodes.

[0026] Each edge is assigned an attribute that describes the type of connection, distance, or approximate travel time, thereby generating a network graph with attributes, namely the location topology graph.

[0027] Preferably, the natural language description data is parsed to extract the event names, times of occurrence, locations involved, and individual or group numbers of livestock and poultry describing the breeding activities, forming structured text records, including:

[0028] Each natural language description is segmented and part-of-speech tagged using a natural language understanding model.

[0029] Based on a predefined aquaculture event vocabulary, keywords representing event names in sentences are matched and located;

[0030] Identify time-related words or phrases in sentences and convert them into standard timestamps in a uniform format;

[0031] Identify words in sentences that indicate location or enclosure number as location information;

[0032] Identify the ear tag numbers, batch numbers, or group characteristic descriptions of livestock and poultry appearing in sentences as livestock and poultry identifiers;

[0033] The extracted event name, standard timestamp, location information, and livestock identifier are combined into a structured data object, which is then used as a structured text record.

[0034] Preferably, for each independent breeding stage, from all event entries belonging to the breeding stage, the core operation types frequently performed within the breeding stage, material information used, and equipment status involved are summarized as a stage behavior summary, including:

[0035] Filter out all event entries in the dynamic event library whose start and end times fall within the time interval of the current breeding stage;

[0036] Statistically analyze the frequency of different event types in these event entries, and identify the event types with a frequency higher than a set threshold as the core operation types of the breeding stage;

[0037] Extract material information such as feed name, drug name, and vaccine type from the ancillary information of all event entries, and record the total amount or frequency of their use;

[0038] Extract the operating parameters or status codes of temperature control equipment, ventilation equipment, and feeding equipment from the ancillary information of all event entries;

[0039] The core operation types, material information, and equipment status are summarized to generate a concise text description, which is the summary of the stage behavior.

[0040] Preferably, the analysis of contextual information about location descriptions in natural language description data, and the movement trajectories of livestock or personnel in video stream data, infers whether there is a direct physical connection or logical jurisdictional relationship between any two locations, including:

[0041] In natural language descriptions, search for phrases indicating location transitions. When two location names appear sequentially in the same transition phrase, infer that there is a direct physical connection between the two locations.

[0042] In natural language descriptions, look for phrases indicating a subordinate relationship. When one location name is described as a component of another location name, infer that there is a logical jurisdictional relationship between the two locations.

[0043] In video stream data, tracking the movement of a specific individual animal or its handler across consecutive frames, when the image features change continuously from the background environment representing one location to the background environment representing another location, infers a direct physical connection between the two locations.

[0044] Based on the inferences from the text and video, when any source confirms the existence of a relationship, the connection or jurisdictional relationship between the two locations is confirmed.

[0045] Preferably, state change parameters of all relevant livestock and poultry individuals during the breeding stage are extracted from the dynamic event database. These state change parameters include changes in activity level, changes in feeding frequency, and weight gain. These parameters are then aggregated and calculated, including:

[0046] Identify all livestock and poultry individuals participating in the current breeding stage based on the livestock and poultry identifiers in the event entries;

[0047] For each individual livestock and poultry, the event that records its key physiological or behavioral indicators is searched from the dynamic event database, and the indicator values ​​and corresponding time points are extracted.

[0048] Calculate the difference in indicators for each individual livestock and poultry at the beginning and end of the current breeding stage, and use it as the change in the state of the individual livestock and poultry.

[0049] By summing, averaging, or calculating the distribution statistics of the state changes of all relevant livestock and poultry individuals, the aggregated value of the state change parameters representing the entire group during the breeding stage is obtained.

[0050] Preferably, by combining the location topology map with the breeding stage sequence, a set of breeding operation instructions that should be executed in the subsequent breeding process is derived for a specific livestock individual or group, including:

[0051] Based on the characteristic representation of the last stage in the breeding stage sequence, predict the target quality status of the next breeding stage;

[0052] Query the location topology map to obtain the current location node of the target livestock and poultry, as well as the feasible path to the target location node that meets the requirements of the next stage of breeding;

[0053] Based on the feasible path and the historical operation efficiency recorded in the dynamic event database, a time window is planned for each location transfer and each operation at each dwell point in the path.

[0054] The specific operations to be performed, the location where the operations occur, and the start and end time windows of the operations are bound together to generate a series of aquaculture operation instructions with spatiotemporal constraints. All instructions constitute the aquaculture operation instruction set.

[0055] Preferably, based on the characteristic representation of the last stage in the breeding stage sequence, the target quality state of the next breeding stage is predicted, including:

[0056] The aquaculture stage sequence is input into the time-series prediction model, which has been trained using historical aquaculture stage sequences and subsequent actual stage data.

[0057] The breeding stage sequence is processed using a time-series prediction model, and a prediction vector with the same dimension as the feature representation of the breeding stage is output.

[0058] The numerical values ​​of each dimension in the prediction vector are analyzed and mapped back to specific quality status descriptions, including the expected weight range, expected health score, and expected behavior pattern. These descriptions together constitute the target quality status for the next breeding stage.

[0059] Compared with the prior art, the beneficial effects of the present invention are:

[0060] By jointly analyzing parallel video stream data and natural language description data, rather than processing them separately, the system can automatically identify and associate information fragments scattered across different modalities. Video content provides visual evidence and precise timestamps of events, while text descriptions supplement the semantics of the events, the specific individuals involved, and the target location. This multimodal analysis technology transforms raw, unstructured observation records into a structured, dynamic sequence of events containing time, location, subjects, and actions. The resulting dynamic event database is traceable and searchable; any feeding, immunization, or abnormal behavior can be precisely pinpointed to the specific time, the specific livestock individual, and the specific location, achieving a shift from passive recording to active perception and structured cognition.

[0061] This approach analyzes the relationships between various locations mentioned in unstructured records, going beyond simple identifier storage to proactively identify and formally define the spatial logical relationships between locations. It weaves discrete, point-like location information into a coherent, networked location topology map. This map digitizes the physical space of the aquaculture environment into a computable and reasonable model. Location information is no longer isolated labels but becomes crucial context in the generation of operational instructions. Based on this map, the system can understand the physical feasibility of operations, thus providing underlying spatial logic support for the automated scheduling and risk management of aquaculture operations. Attached Figure Description

[0062] Figure 1 This is a schematic diagram illustrating the working principle of the livestock and poultry breeding whole-process quality information management method based on multimodal AI described in this invention.

[0063] Figure 2 A flowchart for creating a dynamic event library;

[0064] Figure 3 A flowchart for forming the sequence of aquaculture stages;

[0065] Figure 4 A scatter plot for evaluating the continuity of video trajectories in the construction of a topological map of livestock and poultry breeding locations;

[0066] Figure 5 Line graph showing the efficiency of livestock and poultry farming operation instructions. Detailed Implementation

[0067] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0068] Please see Figure 1 This invention provides a method for managing the quality information of the entire livestock and poultry breeding process based on multimodal AI. The method includes: collecting unstructured records containing video stream data and natural language description data; parsing the video stream data and natural language description data to create a dynamic event database containing breeding events, times, locations, and identification of participating livestock and poultry; using the dynamic event database, the natural division of the breeding process is used to segment the continuous timeline into multiple breeding stages, with the core quality status information of each stage represented by a feature representation, thus forming a breeding stage sequence; parsing all physical or logical locations mentioned in the unstructured records, determining the inclusion, adjacency, or path reachability relationships between locations, and constructing a location topology map describing the spatial connectivity within the breeding environment; and combining the location topology map with the breeding stage sequence to derive the set of breeding operation instructions to be executed in subsequent breeding processes for specific livestock and poultry individuals or groups.

[0069] Example 1: See Figure 2The video stream data is scanned frame by frame, identifying individual livestock, handlers, equipment, and specific behaviors. The identification results are associated with corresponding timestamps to generate time-stamped visual feature segments. Natural language description data is parsed, extracting the event name, time of occurrence, location, and individual or group livestock IDs to form structured text records. A natural language understanding model performs word segmentation and part-of-speech tagging on each description. Based on a predefined livestock event vocabulary, keywords representing event names are matched and located. Time-related words or phrases are identified and converted into standardized timestamps. Words indicating location or pen numbers are identified as location information. Livestock ear tag numbers, batch numbers, or group characteristic descriptions are identified as livestock identifiers. The extracted event name, standard timestamp, location information, and livestock identifiers are combined into a structured data object, serving as a structured text record. Time-stamped visual feature fragments are spatiotemporally aligned with structured text records. When the events identified in the visual feature fragments match the events described in the text records in terms of time and location, the two information are merged. The merged event information is arranged in chronological order of occurrence and stored in a uniform entry format. Each entry contains at least the event type, start time, end time, associated location, and livestock identifier, thus creating a dynamic event database.

[0070] In practice, the processing of video stream data and natural language description data is achieved through an integrated workflow aimed at building a structured dynamic event library from multimodal unstructured records. Video stream data originates from fixed cameras or mobile recording devices deployed within the breeding area, while natural language description data comes from logs or speech-to-text records entered by breeders via handheld terminals. An example scenario involves a fattening pig pen; video stream data records pig activities, breeder feeding and inspection, etc., while natural language description data includes text records submitted by breeders during the same time period. The video stream data is scanned and analyzed frame-by-frame using a pre-trained image recognition model. This model can identify individual pig ear tags, breeder identification, automatic feeder equipment, and specific behaviors such as "eating," "lying down," and "fighting." The analysis results for each frame, including the identified object category, object identifier, and action label, are associated with the precise timestamp corresponding to that frame. The system segments all consecutive frame sequences containing recognition results, generating visual feature segments with start and end timestamps.

[0071] In some embodiments, the text parsing task of the natural language description data is undertaken by a natural language understanding model. The natural language understanding model performs basic processing such as word segmentation and part-of-speech tagging on each input text description. Based on a predefined and continuously updated livestock event vocabulary, the natural language understanding model locates keywords in sentences; for example, it matches and locates the event name "vaccination" from "vaccinated". Time-related words or phrases in the sentence, such as "9:00 AM", are identified and converted into a standard timestamp in the format "2023-10-26 09:00:00" via a time normalization module. Words indicating location, such as "A3 column", are extracted as location information. Livestock identification information, such as "batch P20231001", is identified and extracted by matching preset numbering rules or querying known group characteristics from a database.

[0072] In practice, the spatiotemporal alignment verification module is responsible for associating visual feature fragments with structured text records. This module compares the event type, time information, and location information provided by both. When a "feeder distributing feed" event identified in a visual feature fragment overlaps with a "feeding" event described in a structured text record within a time window, and both location information points to the "A3 column," the verification is considered successful. After successful verification, the information from both sources is merged. The merged information entry includes the precise start and end times and behavioral detail snapshots from the visual source, and the explicit event classification, operator intent, and complete batch number from the text source. Optionally, for data that fails the spatiotemporal alignment verification, the system marks it as an entry to be verified and temporarily stores it in a buffer area, awaiting subsequent supplementary information or manual confirmation, rather than discarding it directly to ensure data integrity. The aligned and merged event information is sorted according to global time order and stored in a dynamic event database in a unified entry format. Each entry must include an event type, start time, end time, associated location, and livestock identification field. In practice, the dynamic event library is stored using a time-indexed database to support efficient queries based on time ranges. Its storage format ensures that each record is independent and contains all necessary fields.

[0073] Example 2: See Figure 3All event entries in the dynamic event database are traversed. Based on pre-defined breeding process knowledge, a continuous occurrence interval of a specific type of event is defined as an independent breeding stage. For each independent breeding stage, the core operation types frequently executed within the breeding stage, the material information used, and the equipment status involved are summarized from all event entries belonging to that stage as a stage behavior summary. All event entries in the dynamic event database whose start and end times fall within the current breeding stage's time interval are filtered. The frequency of different event types in these event entries is statistically analyzed, and event types with a frequency exceeding a set threshold are identified as the core operation types of the breeding stage. Material information such as feed names, drug names, and vaccine types are extracted from the ancillary information of all event entries, and their total amount or frequency of use is recorded. The operating parameters or status codes of temperature control equipment, ventilation equipment, and feeding equipment are extracted from the ancillary information of all event entries. The core operation types, material information, and equipment status are summarized to generate a concise text description, which is the stage behavior summary. State change parameters of all relevant livestock and poultry individuals within the breeding stage are extracted from the dynamic event database. These parameters include changes in activity level, feeding frequency, and weight gain, and are aggregated and calculated. The stage behavior summary is combined with the aggregated state change parameters and encoded into a fixed-dimensional numerical vector. This numerical vector represents the feature representation of the core quality status information of the breeding stage. All breeding stages are arranged in chronological order, and the corresponding feature representations are connected sequentially to form the breeding stage sequence.

[0074] In practice, the formation of the breeding stage sequence is accomplished through a series of coherent data processing steps based on structured records in a dynamic event database. An example scenario involves a broiler breeding batch. The dynamic event database stores all event entries from chicks entering the coop to adult chickens being sold, arranged chronologically. Event types include "temperature control," "feeding," "water management," "light control," "disease monitoring," and "weight sampling." The stage segmentation module traverses all event entries in the dynamic event database and defines a continuous interval of specific event types as an independent breeding stage based on pre-defined breeding process knowledge. This pre-defined breeding process knowledge clearly defines the combination of event types and duration ranges corresponding to stages such as "brooding period," "growth period," and "fattening period." For example, when the system detects a pattern of "temperature control" and "feeding" events dominating for several consecutive days, accompanied by periodic "weight sampling," it identifies this continuous time interval as the "growth period" based on the knowledge base, thus dividing the continuous timeline into multiple breeding stages with clear boundaries.

[0075] In some embodiments, for each defined independent breeding stage, the generation of a stage behavior summary is achieved by statistically analyzing all event entries belonging to that stage. The system filters out all event entries in the dynamic event library whose start and end times fall within the time interval of the current breeding stage. For example, for the "growth period," the system extracts all event records within that time period. The frequency of different event types in these event entries is counted, and event types with a frequency higher than a set threshold are identified as the core operation types of the breeding stage. In the example of the "growth period," "automatic feeding" and "circulating ventilation" may be identified as core operation types. From the ancillary information of all event entries, material information such as feed name, medicine name, and vaccine type is extracted, and the total amount or frequency of their use is recorded. From the ancillary information of all event entries, the operating parameters or status codes of temperature control equipment, ventilation equipment, and feeding equipment are extracted. The core operation types, material information, and equipment status are summarized to generate a concise text description as a stage behavior summary.

[0076] It is understandable that extracting and aggregating the state change parameters of all relevant livestock and poultry individuals within the breeding stage from a dynamic event database is a quantitative analysis process. Based on the livestock and poultry identifiers in the event entries, all livestock and poultry individuals participating in the current breeding stage are identified, for example, the entire flock identified by the batch number "B-2023-09". For each livestock and poultry individual, events recording their key physiological or behavioral indicators are searched from the dynamic event database, and the indicator values ​​and corresponding time points are extracted. For example, the weight value of individual chickens is extracted from the "weight sampling" event, and activity frequency data is extracted from the "activity monitoring" event. The difference in indicators for each livestock and poultry individual at the beginning and end of the current breeding stage is calculated as the state change of the individual. The state change quantities of all relevant livestock and poultry individuals are aggregated to obtain the aggregated value of the state change parameters representing the entire group during the breeding stage. The aggregation calculation can be performed using summation, averaging, or calculating distribution statistics. One method is used to calculate the average weight gain of the group. Here are some examples:

[0077] ;

[0078] in: This indicates the total number of relevant livestock and poultry individuals during the breeding stage. Indicates the first The weight of each individual at the beginning of the phase. Indicates the first The weight of each individual at the end of the phase. This represents the average weight gain of the group. Similar aggregation methods are applied to parameters such as changes in activity level and changes in eating frequency.

[0079] In practice, the stage behavior summary and the aggregated state change parameters are combined into a fixed-dimensional numerical vector through an encoding process. The encoding process converts the text-based stage behavior summary into a numerical vector using a pre-trained word embedding model, while simultaneously standardizing the aggregated numerical values ​​of the state change parameters. The two are then concatenated into an intermediate vector, which is then mapped to a pre-defined fixed dimension through a linear transformation layer. For example, the stage behavior summary is converted into a 100-dimensional embedding vector, and the aggregated state change parameters are used as a 3-dimensional vector; after concatenation, a 103-dimensional vector is obtained. Finally, a linear transformation outputs a 50-dimensional numerical vector, which represents the feature representation of the core quality state information of the breeding stage.

[0080] Optionally, all breeding stages are arranged chronologically, and their corresponding feature representations are sequentially concatenated to form a breeding stage sequence. The system arranges all breeding stages in timestamp order and sequentially concatenates the 50-dimensional feature representations corresponding to each stage into a longer vector. Assuming there are three stages, the concatenation results in a 150-dimensional vector sequence, which fully represents the temporal evolution of the quality state in the breeding process. The breeding stage sequence is stored in a dedicated sequence database for subsequent process analysis and instruction generation modules to access. In some embodiments, the construction of the breeding stage sequence is periodic; whenever a breeding stage is confirmed to be completed, the system automatically triggers the feature representation calculation and sequence update operation for the new stage.

[0081] Example 3: A unique set of location names is extracted from all event entries in the dynamic event database, and each location name is standardized. Contextual information about location descriptions in the natural language description data is analyzed, and the movement trajectories of livestock or personnel in the video stream data are analyzed. Whether a direct physical connection or logical jurisdictional relationship exists between any two locations is inferred. Phrases indicating location transfers in the natural language description are searched; when two location names appear sequentially in the same transfer phrase, a direct physical connection between the two locations is inferred. Phrases indicating subordinate relationships in the natural language description are searched; when one location name is described as a component of another location name, a logical jurisdictional relationship between the two locations is inferred. In the video stream data, the movement of specific livestock individuals or breeders across consecutive frames is tracked; when their image features continuously change from the background environment representing one location to the background environment representing another location, a direct physical connection between the two locations is inferred. The inference results from the text and video are combined; when any source confirms a relationship, a connection or jurisdictional relationship between the two locations is confirmed. Each standardized location name is abstracted as a node. If a direct connection or jurisdictional relationship exists between two locations, an edge is established between the two corresponding nodes. Each edge is assigned attributes that describe the type of connection, distance, or approximate travel time. A network graph with these attributes is then generated, which is the location topology graph.

[0082] In practical implementation, the construction of the location topology map originates from the systematic analysis and relationship inference of the location information mentioned in all event entries in the dynamic event database. An example scenario is a multi-level, vertically integrated egg-laying hen house. The events recorded in the dynamic event database are associated with multiple physical or logical location names such as "third-floor east feeding area," "second-floor vaccination point," "central egg collection belt entrance," "bottom-floor manure treatment room," and "feed tower." The system first extracts a unique set of location names from all event entries in the dynamic event database and standardizes each location name, standardizing "bottom-floor manure treatment room" to "manure treatment room," thereby eliminating descriptive ambiguity and forming a standardized location list.

[0083] In some embodiments, analyzing the contextual information about location descriptions in natural language description data, and the movement trajectories of livestock or personnel in video stream data, is the primary means of inferring the relationship between any two locations. Phrases indicating location transfer are searched in the natural language description data. When two location names appear sequentially in the same transfer phrase, a direct physical connection between the two locations is inferred. For example, from the record "transferring chickens from the brooder room to the pullet house," a direct physical connection between "brooder room" and "pullet house" can be inferred. Phrases indicating subordinate relationships are searched in the natural language description data. When one location name is described as a component of another location name, a logical jurisdictional relationship between the two locations is inferred. For example, from the record "checking the waterline in the three-layer feeding area," a jurisdictional relationship can be inferred that "waterline" is a logical component of "three-layer feeding area." In video stream data, the movement of a specific individual animal or a feeder across consecutive frames is tracked. When the image features change continuously from the background environment representing one location to the background environment representing another location, a direct physical connection between the two locations is inferred. For example, a sequence of images is tracked through video to show a feeder walking from the "feed preparation room" to the "central aisle".

[0084] It is understandable that the inferences from the text and video above require comprehensive judgment. When any source confirms the existence of a relationship, the connection or jurisdictional relationship between the two locations is confirmed. The system assigns a confidence score to each inference, weighted based on the reliability of the information sources. For inferences from different information sources, a fusion rule is used for judgment. A rule is used to calculate the comprehensive confidence score of the relationship. Here are some examples:

[0085] ;

[0086] in: Indicates location inference based on natural language description data. With position The confidence level of the relationship is between 0 and 1, where 1 indicates that the transfer or subordination relationship is clearly described in the text, and 0 indicates that no relevant description was found. This indicates the confidence level of inferring a physical connection between the two based on video stream data; its value depends on the continuity and clarity of the movement trajectory in the video. and It is a coefficient that adjusts the weights of text and video evidence, and satisfies... ; Indicates the overall confidence level, when Exceeding the preset threshold At that time, the location is confirmed. With position There is a relationship between them.

[0087] In practical implementation, the graph construction module abstracts each standardized location name into a graph node. If a comprehensive judgment confirms a direct connection or jurisdictional relationship between two locations, an edge is established between the corresponding two nodes. Each edge is assigned attributes describing the connection type, distance, or approximate travel time. For example, an edge between the "brooding room" node and the "young chicken coop" node is labeled with the attributes "Connection type: physical passage, distance: 15 meters, estimated travel time: 2 minutes"; an edge between the "three-layer feeding area" node and the "water line" node is labeled with the attribute "Connection type: logical jurisdiction". All nodes and attributed edges together constitute an attributed network graph, i.e., a location topology graph. Optionally, the location topology graph can be stored and visualized in the form of a graph database. The graph not only records the binary relationships between locations but can also calculate the reachable and optimal paths between any two locations using graph path search algorithms. In some embodiments, the map is incrementally maintained based on continuous updates of the dynamic event library. When new location names or location relationships are identified, the corresponding nodes and edges are added or updated, enabling the location topology map to dynamically reflect changes in the spatial layout of the aquaculture environment.

[0088] See Figure 4 This is a scatter plot of video trajectory continuity ratings used in the construction of a livestock and poultry farming location topology map. It assesses the clarity of livestock / personnel movement trajectories in different video clips and is one of the core bases for location relationship inference. High scores are concentrated above 0.8, indicating that the trajectory quality of most video clips is good; medium scores are distributed in the 0.3-0.8 range, with a moderate proportion, reflecting slight interference in some videos; low scores are only present in a small number, indicating a low proportion of extremely blurry trajectories. This type of chart is mainly used to quantify the reliability of video evidence, helping the system determine the weight of video trajectories in location relationship inference, and is one of the key reference bases for "multi-source information fusion" in location topology map construction.

[0089] Example 4: State change parameters of all relevant livestock and poultry individuals during the rearing stage are extracted from the dynamic event database. These parameters include changes in activity level, feeding frequency, and weight gain. These parameters are then aggregated and calculated. Based on the livestock and poultry identifiers in the event entries, all livestock and poultry individuals participating in the current rearing stage are identified. For each livestock and poultry individual, events recording their key physiological or behavioral indicators in the dynamic event database are searched, and the indicator values ​​and corresponding time points are extracted. The difference in indicators for each livestock and poultry individual at the beginning and end of the current rearing stage is calculated as the state change quantity of that individual. The state change quantities of all relevant livestock and poultry individuals are summed, averaged, or their distribution statistics are calculated to obtain the aggregated value of the state change parameters of the entire group during the rearing stage.

[0090] In practical implementation, extracting and aggregating the state change parameters of all relevant livestock and poultry individuals within the breeding stage from the dynamic event database is a specific data processing flow. This flow is applied to a defined breeding stage. An example scenario involves a 30-day "fattening period" for beef cattle. The dynamic event database records event entries related to individual cattle within this stage. Based on the livestock and poultry identifiers in the event entries, all livestock and poultry individuals participating in the current breeding stage are identified. In this example, the system determines that the beef cattle participating in the "fattening period" are ten cattle numbered from C001 to C010 by querying the ear tag numbers appearing in the event entries.

[0091] In some embodiments, retrieving events from a dynamic event database that record key physiological or behavioral indicators for each livestock individual and extracting the indicator values ​​and corresponding time points is a precise data retrieval process. For each beef cattle individual, the system retrieves entries from the dynamic event database with event types of "weight measurement," "activity record," or "feed intake record," and whose livestock identifier matches the individual's number. For example, for individual C001, the system retrieves two "weight measurement" events, recorded on the start date (time point T_start) and end date (time point T_end) of the "fattening period," respectively, with extracted weight values ​​of W_C001_start and W_C001_end; it also retrieves multiple "activity record" events, which record daily activity counts. It can be understood that calculating the difference in indicators between the start and end of the current rearing stage for each livestock individual, as the state change of the individual, is the basis for subsequent aggregation. For the weight indicator, the individual state change is the weight value at the end minus the weight value at the beginning. For behavioral indicators such as activity level or feeding frequency, which may have multiple records, it is necessary to first calculate the daily average value within the period, and then calculate the change. Refer to Table 1, which shows the calculation of the change in the condition of an individual beef cattle during the "fattening period".

[0092] Table 1: Calculation of changes in the condition of individual beef cattle during the "fattening period"

[0093]

[0094] In practice, a summary analysis step involves summing, averaging, or calculating the distribution statistics of the state changes of all relevant livestock and poultry individuals to obtain aggregated values ​​of state change parameters representing the entire group during the rearing stage. The system reads the state changes of each individual calculated as shown in the table above. Aggregating these sets and calculating the average yields the average weight gain and average activity level change of the group; calculating the standard deviation reveals the dispersion of the group's weight gain. An aggregated value for calculating group state change parameters, such as the average weight gain, is used. and the coefficient of variation of group weight gain Here are some examples:

[0095] ;

[0096] ;

[0097] in: This indicates the total number of relevant livestock and poultry individuals during the breeding stage. Indicates the first The amount of weight gain for each individual. This represents the total weight gain of all individuals. standard deviation This indicates the average weight gain of the group. The coefficient of variation represents the amount of weight gain in a group and is used to measure the consistency of weight gain among individuals within the group.

[0098] Optionally, the aggregation calculation is not limited to the average and coefficient of variation mentioned above; other statistics can also be used depending on management needs. In some embodiments, the system calculates the median, maximum, and minimum weight gain to understand the distribution range, or calculates the proportion of individuals with negative activity changes to assess the group's stress status. All calculated aggregated values ​​of state change parameters, such as average weight gain, coefficient of variation of weight gain, and average activity change, will be encapsulated into a structured dataset for subsequent use in forming a characteristic representation of the breeding stage. It can be understood that this aggregation process transforms dispersed individual indicators into core parameters representing the overall state of the group.

[0099] Example 5: The feature representation of the last stage in the breeding stage sequence is used to predict the target quality state of the next breeding stage. The breeding stage sequence is input into a time-series prediction model, which has been trained using historical breeding stage sequences and subsequent actual stage data. The time-series prediction model processes the breeding stage sequence and outputs a prediction vector with the same dimensions as the breeding stage feature representation. The values ​​of each dimension in the prediction vector are parsed and mapped back to specific quality state descriptions, including the expected weight range, expected health score, and expected behavior pattern. These descriptions together constitute the target quality state of the next breeding stage. The location topology map is queried, the current location node of the target livestock is obtained, and feasible paths to the target location node that meets the breeding requirements of the next stage are obtained. Based on the feasible paths and the historical operation efficiency recorded in the dynamic event database, the operation planning time window for each location transfer and each dwelling point in the path is planned. The specific operations to be executed, the location where the operation occurs, and the start and end time windows of the operation are bound together, and breeding operation instructions with spatiotemporal constraints are generated. All instructions constitute a breeding operation instruction set.

[0100] In practical implementation, the set of breeding operation instructions for specific livestock individuals or groups is derived by combining location topology maps with breeding stage sequences, involving a serialization process of prediction, path planning, and instruction binding. An example scenario focuses on a broiler flock in the "mid-growth stage," where the breeding stage sequence records the feature representations of several stages from the "brooding period" to the current "mid-growth stage." The time-series prediction model receives the breeding stage sequence as input, which is a list of feature vectors arranged in chronological order. The time-series prediction model has been trained using historical breeding stage sequences and subsequent actual stage data, enabling it to infer trends for subsequent stages from historical sequence patterns. The time-series prediction model processes the input breeding stage sequence and outputs a prediction vector P with the same dimension as the breeding stage feature representation. The values ​​of each dimension in the prediction vector P are analyzed and mapped back to specific quality status descriptions. For example, the value of one dimension is mapped to "expected weight range: 2.1-2.3 kg", the value of another dimension is mapped to "expected health score > 85", and the values ​​of other dimensions are mapped to "expected daily activity level not less than X times". These descriptions together constitute the target quality status of the next breeding stage.

[0101] In some embodiments, querying the location topology map to obtain feasible paths is a prerequisite for planning physical movement. The current location node of the target livestock is obtained by querying the latest location record containing the group's identifier in the dynamic event database. The target location node that meets the requirements of the next stage of breeding is determined based on the breeding procedures and the predicted target quality status. The system queries the graph database storing the location topology map to request the calculation of all feasible paths from the "No. 3 Hearing Ward B Area" node to the "Testing Center" node. Edge attributes in the location topology map, such as connection type and estimated travel time, are used to evaluate the feasibility and time cost of each path. The system may select the path with the shortest total estimated travel time as the feasible path for this execution.

[0102] It is understandable that planning time windows for each location transfer and operation at each stop point within a feasible path, based on historical operational efficiency recorded in the dynamic event database, is a scheduling process based on historical data. The historical operational efficiency recorded in the dynamic event database includes the average time required to complete similar transfer tasks and the historical average time spent performing a specific operation at a specific location. The system plans time windows for each segment of movement within the feasible path, such as "transfer from Area B of Incubation Dormitory No. 3 to the central corridor, planned time window: 09:00-09:10". The system also plans time windows for the "disease testing" operation to be performed at the target location, the "Testing Center". The duration of this window is determined based on the historical average time for completing similar testing operations, such as "perform disease testing at the Testing Center, planned time window: 09:15-09:45". The planning process must ensure that the movement window and operation window are temporally continuous and conflict-free, and a time window is used to calculate the operation end time. Here are some examples:

[0103] ;

[0104] in: Indicates the start time of the transfer. This indicates the total number of transfer segments included in the feasible path. This indicates that the estimate based on historical data is obtained through the [number]th [period]. The required duration for each road segment This indicates the historical average time taken to perform this specific operation at the target location. This indicates the planned end time of the operation. This formula is used to deduce the transfer start time backward from the operation requirements, or to deduce the operation completion time forward from the transfer end time.

[0105] Optionally, the generated set of aquaculture operation instructions will be sent to the control terminal of the breeder or automated equipment via a message queue or task management interface. In some embodiments, the instruction set will include a planar schematic diagram of the relevant area in the location topology map and path navigation guidance. It can be understood that the prediction results of the time series prediction model, the path search of the location topology map, and historical operation efficiency data jointly determine the spatiotemporal parameters of the final generated instruction set. The instruction set is dynamically generated. When the actual execution progress deviates from the plan, or when new state information is injected into the dynamic event library, the system will re-execute the derivation process to update the subsequent instruction set, thereby achieving dynamic and precise guidance of the aquaculture process.

[0106] See Figure 5This is a line chart showing the efficiency of livestock and poultry farming operation instructions. It displays the cumulative trends of "completed, delayed, and total number of instructions" during the instruction execution process and is one of the core efficiency monitoring charts in the instruction derivation stage. The gradually narrowing gap between the number of completed instructions and the total number of instructions indicates that execution efficiency improves over time; the continuous upward trend reflects the increasing demand for instructions as the farming process progresses; the slow growth in the number of delayed instructions, representing a low proportion of the total, indicates good overall timeliness of instruction execution. This type of chart is primarily used to assess the efficiency and progress of farming operation instructions, helping managers identify execution bottlenecks and optimize the time window planning for subsequent instructions. It is a key monitoring tool for ensuring the farming process proceeds as planned.

[0107] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0108] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for quality information management throughout the entire livestock and poultry farming process based on multimodal AI, characterized in that: The processing includes the following stages: The collected unstructured records of livestock and poultry farming are analyzed and transformed. The unstructured records include video stream data and natural language description data. By parsing the video stream data and natural language description data, a dynamic event database containing farming events, time, location and identification of participating livestock and poultry is created. Based on the dynamic event library, the continuous timeline is divided into multiple breeding stages according to the natural division of the breeding process, and a feature representation that can represent the core quality status information of each breeding stage is extracted to form a breeding stage sequence. All physical or logical locations mentioned in the unstructured records are parsed to determine the inclusion, adjacency, or path reachability relationships between locations, and a location topology map describing the spatial connectivity within the aquaculture environment is constructed. By combining the location topology map with the breeding stage sequence, a set of breeding operation instructions that should be executed in the subsequent breeding process can be derived for a specific livestock individual or group.

2. The method for managing the quality information of the entire livestock and poultry breeding process based on multimodal AI according to claim 1, characterized in that, The collected unstructured records of livestock and poultry farming are analyzed and transformed. These unstructured records include video stream data and natural language description data. By parsing the video stream data and natural language description data, a dynamic event database is created, containing farming events, times, locations, and identifiers of participating livestock and poultry. The video stream data is scanned frame by frame to identify individual livestock, poultry, feeders, equipment, and specific behaviors in the scene. The identification results are associated with the corresponding timestamps to generate visual feature segments with time stamps. Natural language description data is parsed to extract event names, times of occurrence, locations involved, and individual or group numbers of livestock and poultry that describe breeding activities, forming structured text records; The time-stamped visual feature fragments are spatiotemporally aligned with the structured text records. When the events identified in the visual feature fragments are consistent with the events described in the text records in terms of time and location, the two information are merged. The merged event information is arranged in chronological order of occurrence and stored in a uniform entry format. Each entry contains at least the event type, start time, end time, associated location, and livestock identifier, thereby creating the dynamic event library.

3. The method for managing the quality information of the entire livestock and poultry breeding process based on multimodal AI according to claim 1, characterized in that, Based on the dynamic event database, the continuous timeline is divided into multiple breeding stages according to the natural division of the breeding process. For each breeding stage, a feature representation that can represent the core quality status information of that stage is extracted, forming a breeding stage sequence, including: Traverse all event entries in the dynamic event library, and define a continuous occurrence interval of a specific type of event as an independent breeding stage based on the preset breeding process knowledge; For each independent breeding stage, from all event entries belonging to the breeding stage, the core operation types frequently performed within the breeding stage, the material information used, and the equipment status involved are summarized as a stage behavior summary. Extract the state change parameters of all relevant livestock and poultry individuals during the breeding stage from the dynamic event database. The state change parameters include changes in activity level, changes in feeding frequency, and weight gain. Then, aggregate and calculate these parameters. The stage behavior summary and the aggregated state change parameters are combined and encoded into a fixed-dimensional numerical vector, which is the feature representation of the core quality state information of the breeding stage. Arrange all breeding stages in chronological order and connect their corresponding feature representations sequentially to form the breeding stage sequence.

4. The method for managing the quality information of the entire livestock and poultry breeding process based on multimodal AI according to claim 1, characterized in that, The unstructured records are parsed to determine all physical or logical locations mentioned, identifying inclusion, adjacency, or path reachability relationships between these locations. A location topology map describing the spatial connectivity within the aquaculture environment is then constructed, including: Extract a unique set of location names from all event entries in the dynamic event library, and standardize each location name. By analyzing the contextual information about location descriptions in natural language description data and the movement trajectories of livestock or people in video stream data, it can be inferred whether there is a direct physical connection or logical jurisdiction relationship between any two locations. Each standardized location name is abstracted into a node. If it is determined that there is a direct connection or jurisdiction relationship between two locations, an edge is established between the corresponding two nodes. Each edge is assigned an attribute that describes the type of connection, distance, or approximate travel time, thereby generating a network graph with attributes, namely the location topology graph.

5. The method for managing the quality information of the entire livestock and poultry breeding process based on multimodal AI according to claim 2, characterized in that, Natural language description data is parsed to extract event names, times of occurrence, locations involved, and individual or group numbers of livestock and poultry describing farming activities, forming structured text records, including: Each natural language description is segmented and part-of-speech tagged using a natural language understanding model. Based on a predefined aquaculture event vocabulary, keywords representing event names in sentences are matched and located; Identify time-related words or phrases in sentences and convert them into standard timestamps in a uniform format; Identify words in sentences that indicate location or enclosure number as location information; Identify the ear tag numbers, batch numbers, or group characteristic descriptions of livestock and poultry appearing in sentences as livestock and poultry identifiers; The extracted event name, standard timestamp, location information, and livestock identifier are combined into a structured data object, which is then used as a structured text record.

6. The method for managing the quality information of the entire livestock and poultry breeding process based on multimodal AI according to claim 3, characterized in that, For each independent breeding stage, from all event entries belonging to that breeding stage, the core operation types frequently performed within that breeding stage, the material information used, and the equipment status involved are summarized as a stage behavior summary, including: Filter out all event entries in the dynamic event library whose start and end times fall within the time interval of the current breeding stage; Statistically analyze the frequency of different event types in these event entries, and identify the event types with a frequency higher than a set threshold as the core operation types of the breeding stage; Extract material information such as feed name, drug name, and vaccine type from the ancillary information of all event entries, and record the total amount or frequency of their use; Extract the operating parameters or status codes of temperature control equipment, ventilation equipment, and feeding equipment from the ancillary information of all event entries; The core operation types, material information, and equipment status are summarized to generate a concise text description, which is the summary of the stage behavior.

7. The method for managing the quality information of the entire livestock and poultry breeding process based on multimodal AI according to claim 4, characterized in that, Analyze the contextual information about location descriptions in natural language processing data, and the movement trajectories of livestock or people in video stream data, to infer whether there is a direct physical connection or logical jurisdictional relationship between any two locations, including: In natural language descriptions, search for phrases indicating location transitions. When two location names appear sequentially in the same transition phrase, infer that there is a direct physical connection between the two locations. In natural language descriptions, look for phrases indicating a subordinate relationship. When one location name is described as a component of another location name, infer that there is a logical jurisdictional relationship between the two locations. In video stream data, tracking the movement of a specific individual animal or its handler across consecutive frames, when the image features change continuously from the background environment representing one location to the background environment representing another location, infers a direct physical connection between the two locations. Based on the inferences from the text and video, when any source confirms the existence of a relationship, the connection or jurisdictional relationship between the two locations is confirmed.

8. The method for managing the quality information of the entire livestock and poultry breeding process based on multimodal AI according to claim 3, characterized in that, The state change parameters of all relevant livestock and poultry individuals during the breeding stage are extracted from the dynamic event database. These state change parameters include changes in activity level, changes in feeding frequency, and weight gain. These parameters are then aggregated and calculated, including: Identify all livestock and poultry individuals participating in the current breeding stage based on the livestock and poultry identifiers in the event entries; For each individual livestock and poultry, the event that records its key physiological or behavioral indicators is searched from the dynamic event database, and the indicator values ​​and corresponding time points are extracted. Calculate the difference in indicators for each individual livestock and poultry at the beginning and end of the current breeding stage, and use it as the change in the state of the individual livestock and poultry. By summing, averaging, or calculating the distribution statistics of the state changes of all relevant livestock and poultry individuals, the aggregated value of the state change parameters representing the entire group during the breeding stage is obtained.

9. The method for managing the quality information of the entire livestock and poultry breeding process based on multimodal AI according to claim 1, characterized in that, By combining the location topology map with the breeding stage sequence, a set of breeding operation instructions that should be executed in the subsequent breeding process is derived for a specific livestock individual or group, including: Based on the characteristic representation of the last stage in the breeding stage sequence, predict the target quality status of the next breeding stage; Query the location topology map to obtain the current location node of the target livestock and poultry, as well as the feasible path to the target location node that meets the requirements of the next stage of breeding; Based on the feasible path and the historical operation efficiency recorded in the dynamic event database, a time window is planned for each location transfer and each operation at each dwell point in the path. The specific operations to be performed, the location where the operations occur, and the start and end time windows of the operations are bound together to generate a series of aquaculture operation instructions with spatiotemporal constraints. All instructions constitute the aquaculture operation instruction set.

10. The method for managing the quality information of the entire livestock and poultry breeding process based on multimodal AI according to claim 9, characterized in that, Based on the characteristic representation of the last stage in the breeding stage sequence, predict the target quality state of the next breeding stage, including: The aquaculture stage sequence is input into the time-series prediction model, which has been trained using historical aquaculture stage sequences and subsequent actual stage data. The breeding stage sequence is processed using a time-series prediction model, and a prediction vector with the same dimension as the feature representation of the breeding stage is output. The numerical values ​​of each dimension in the prediction vector are analyzed and mapped back to specific quality status descriptions, including the expected weight range, expected health score, and expected behavior pattern. These descriptions together constitute the target quality status for the next breeding stage.

Images