Living body collateral state monitoring method and related device

Abstract

The invention discloses a living body collateral state monitoring method and a related device, and relates to the field of data processing, and the method comprises the steps: obtaining the acoustic features, image features and positioning features of a monitored object, recognizing the scene of the environment where the monitored object is located at present, and determining the target weight distribution matched with the scene. And carrying out weighted fusion on the obtained acoustic features, image features and positioning features of the monitoring object based on the target weight distribution to obtain multi-modal fusion features. And finally, identifying the abnormal state of the monitored object based on the multi-modal fusion feature. According to the method, the complementary effect of the acoustic features, the image features, the positioning features and other dimension features is utilized, the recognition accuracy is improved, and the false recognition rate is reduced. Moreover, the scheme can identify the scene of the current environment and automatically adjust the weight distribution of each modal feature according to the scene characteristics to generate the unique fusion feature vector, and the robustness of the model to the environment is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a living body pledge state monitoring method and related device. BACKGROUND

[0002] As an important financial tool for rural revitalization, the management of living body livestock pledges in the credit field has problems such as easy forgery of identity recognition, lagging dynamic monitoring, and single state evaluation. The traditional management technology has a high error rate in identifying living bodies through a single modal, and the separation of location data and identity data makes it impossible to confirm whether the living body at the location is the pledge, and the abnormal state of the living body cannot be identified in a timely manner. SUMMARY

[0003] In view of the above problems, the present application provides a living body pledge state monitoring method and related device to improve the identification accuracy and the timeliness of abnormal state early warning. The specific scheme is as follows:

[0004] The first aspect of the present application provides a living body pledge state monitoring method, comprising:

[0005] Acquiring the acoustic feature, image feature and positioning feature of the monitoring object in the current monitoring area;

[0006] Determining a target weight distribution matched with the current identification scene based on the light intensity and environmental noise intensity of the environment where the monitoring object is located;

[0007] According to the target weight distribution, the acoustic feature, image feature and positioning feature of the monitoring object are weighted and fused to obtain a multi-modal fusion feature;

[0008] Identifying the abnormal state of the monitoring object based on the multi-modal fusion feature.

[0009] In one possible implementation, the target weight distribution matched with the current identification scene is determined based on the light intensity and environmental noise intensity of the environment where the monitoring object is located, comprising:

[0010] According to the light intensity of the environment where the monitoring object is located, it is identified that the current environment scene is daytime strong light, and the weight coefficient of the image feature is greater than or equal to a first preset value, which is greater than or equal to the sum of the weight coefficients of the acoustic feature and the positioning feature;

[0011] According to the light intensity of the environment where the monitoring object is located, it is identified that the current environment scene is nighttime weak light, and the weight coefficient of the acoustic feature is greater than or equal to a second preset value, which is greater than the sum of the weight coefficients of the image feature and the positioning feature;

[0012] According to the environmental noise intensity of the environment where the monitored object is located, it is identified that the current environment scene is a noisy environment, and the weight coefficient of the acoustic feature is greater than or equal to a third preset value, which is greater than the sum of the weight coefficients of the image feature and the positioning feature.

[0013] In a possible implementation, the acoustic feature, the image feature and the positioning feature of the monitored object are weighted and fused according to the target weight distribution to obtain a multi-modal fusion feature, including:

[0014] Based on the features of a single modality of the monitored object, it is determined whether there is an anomaly;

[0015] If the features of any modality are abnormal, the weight of the abnormal feature is a fourth preset value, which is greater than the sum of the weights of the features without anomaly, and the features of the three modalities are fused according to the re-adjusted weight distribution;

[0016] If the features of a single modality are not abnormal, the features of the three modalities are fused according to the target weight distribution.

[0017] In a possible implementation, the abnormal state of the monitored object is identified based on the multi-modal fusion feature, including:

[0018] If the fusion similarity of the multi-modal fusion feature of the monitored object is less than a first similarity threshold, it is determined that the monitored object is not a target object;

[0019] If the abnormal frequency band energy proportion of the acoustic feature in the multi-modal fusion feature of the monitored object is greater than a third threshold, and the eye temperature of the monitored object is greater than a fourth threshold based on the image feature in the multi-modal fusion feature, it is determined that the health state of the monitored object is abnormal.

[0020] In a possible implementation, the abnormal state of the monitored object is identified based on the multi-modal fusion feature, including:

[0021] Based on the positioning feature in the multi-modal fusion feature, it is identified that the monitored object exceeds a preset range of an electronic fence, and the out-of-bound time length of the monitored object is recorded;

[0022] If the out-of-bound time length is greater than a first preset time length, it is determined that the monitored object has a location out-of-bound anomaly.

[0023] In a possible implementation, the abnormal state of the monitored object is identified based on the multi-modal fusion feature, including:

[0024] identify a number of the monitoring objects in the current monitoring area based on image features or acoustic features in the multi-modal fusion features corresponding to all the monitoring objects in the current monitoring area, to obtain an identified number;

[0025] If a deviation between the identified number and the account number is greater than a preset value, it is determined that the number of monitoring objects in the current monitoring area is abnormal.

[0026] In a possible implementation, the abnormal state includes identity abnormality, health abnormality, location abnormality, and number abnormality; and the method further includes:

[0027] determining a warning level corresponding to the current abnormal state, determining a first-level warning if the current abnormal state is caused by a single modal feature abnormality, and starting a review process;

[0028] determining a second-level warning if the current abnormal state is caused by two modal feature abnormalities, and starting a financial lending institution intervention process;

[0029] determining a third-level warning if the current abnormal state is caused by three modal feature abnormalities, freezing a credit limit, and triggering a multi-modal fusion data on-chain storage process.

[0030] In a possible implementation, after the multi-modal fusion feature is used to identify the abnormal state of the monitoring object, the method further includes:

[0031] obtaining a voiceprint confidence, an image confidence, and a confidence of being in an electronic fence in the multi-modal fusion feature of the monitoring object;

[0032] determining a conflict rate of the multi-modal feature based on the voiceprint confidence, the image confidence, and the confidence of being in the electronic fence corresponding to the monitoring object;

[0033] if the conflict rate is greater than the first threshold value and less than a second threshold value, determining that the abnormal state corresponds to a first-level warning and starting a review process;

[0034] if the conflict rate is greater than or equal to the second threshold value and less than a third threshold value, determining that the abnormal state corresponds to a second-level warning and starting a financial lending institution intervention process;

[0035] if the conflict rate is greater than or equal to the third threshold value, determining that the abnormal state corresponds to a third-level warning, freezing a credit limit, and triggering a multi-modal fusion data on-chain storage process.

[0036] The second aspect of the present application provides a living body collateral state monitoring device, comprising:

[0037] a feature acquisition module configured to acquire acoustic features, image features, and positioning features of monitoring objects in a current monitoring area;

[0038] a weight determination module configured to determine a target weight distribution matched with the current identified scene based on an illumination intensity and an environmental noise intensity of an environment in which the monitored object is located;

[0039] a feature fusion module configured to perform weighted fusion on the acoustic feature, the image feature and the positioning feature of the monitored object according to the target weight distribution, to obtain a multi-modal fusion feature;

[0040] a state identification module configured to identify an abnormal state of the monitored object based on the multi-modal fusion feature.

[0041] A third aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0042] the memory is configured to store a computer program;

[0043] the processor is configured to execute the computer program, so that the electronic device can implement the living pledge state monitoring method according to the first aspect or any implementation manner of the first aspect.

[0044] A fourth aspect of the present application provides a computer program product, comprising computer readable instructions, when the computer readable instructions run on an electronic device, the electronic device implements the living pledge state monitoring method according to the first aspect or any implementation manner of the first aspect.

[0045] A fifth aspect of the present application provides a computer storage medium, the storage medium carries one or more computer programs, when the one or more computer programs are executed by an electronic device, the electronic device can implement the living pledge state monitoring method according to the first aspect or any implementation manner of the first aspect.

[0046] By the technical scheme, the living collateral state monitoring method provided by the application obtains acoustic features, image features and positioning features of a monitoring object, identifies a scene in which the monitoring object currently stays and determines a target weight distribution matched with the scene. The acoustic features, image features and positioning features of the monitoring object obtained are weighted and fused based on the target weight distribution to obtain multi-modal fusion features. Finally, the multi-modal fusion features are used to identify an abnormal state of the monitoring object, such as identity fraud, health abnormality, location out-of-bound, quantity abnormality and the like. The acoustic features can reflect voiceprint characteristics of an animal, the image features reflect visual characteristics of the animal, and the positioning features reflect location information of the animal. The complementary effects of multi-dimensional features are utilized to improve the identification accuracy and reduce the misidentification rate. Moreover, the scheme can identify a current scene and automatically adjust the weight distribution of each modal feature according to the characteristics of the scene to generate a unique fusion feature vector. In addition, the robustness of the model to the environment is improved. For example, in a noisy and weak-light night scene, the multi-modal fusion features can improve the identification accuracy by 32% compared with single-dimensional features. BRIEF DESCRIPTION OF DRAWINGS

[0047] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent as various embodiments thereof are described in conjunction with the following drawings, in which like reference numerals refer to like elements throughout the drawings. It should be understood that the drawings are schematic and elements depicted are not necessarily to scale.

[0048] Figure 1 A structural schematic diagram of a living collateral state monitoring system provided by the application;

[0049] Figure 2 A flowchart of a living collateral state monitoring method provided by the application;

[0050] Figure 3 A schematic diagram of a multi-modal feature extraction and fusion process provided by the application;

[0051] Figure 4 A flowchart of an abnormal state identification process provided by the application;

[0052] Figure 5 A flowchart of a hierarchical early warning process provided by the application;

[0053] Figure 6 A structural schematic diagram of a living collateral state monitoring device provided by the application;

[0054] Figure 7 A structural schematic diagram of an electronic device provided by the application. DETAILED DESCRIPTION

[0055] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0056] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0057] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0058] Currently, in the management of live collateral in the credit field, the identity of live collateral is marked by ear tags or identification chips. However, ear tags or identification chips are easy to fall off and forge. Furthermore, the management of live collateral uses positioning collars to locate the live collateral, but this method leads to a separation between the location data and the identity data of the live collateral, making it impossible to determine whether the animal at that location is collateral. In addition, traditional management methods only record the animal's movement trajectory and cannot combine the animal's behavioral characteristics to analyze its health status, which means that dynamic monitoring is lagging and status assessment is limited.

[0059] To address the problems existing in the management of live collateral in the credit field, this application provides a method for monitoring the status of live collateral. This method integrates data from multiple modalities, including voiceprint, vision, and location, to construct a live identification system with unique biometric authentication, solving the problems of easy detachment and forgery of traditional ear tags / identity chips. A cross-modal status correlation model is established to achieve real-time early warning of abnormal states such as health anomalies, out-of-bounds location, and quantity changes in live collateral.

[0060] Let's combine the following... Figure 1 The structure of the live collateral status monitoring system provided in the embodiments of this application is described.

[0061] like Figure 1 As shown, the system may include: a sound acquisition device, a visual acquisition device, a positioning and tracking device, a multimodal fusion engine, and a risk decision-making module; wherein, the multimodal fusion engine and the risk decision-making module may be deployed on the same device, such as a cloud device.

[0062] The sound collecting device is used for collecting sound signals in the environment, and can be a microphone array.

[0063] The visual collecting device is used for collecting images or videos, and a dual-spectrum camera can be used. The dual-spectrum camera can simultaneously obtain images / videos of two channels of high-definition thermal imaging and visible light images. The infrared channel camera can capture the thermal radiation of an object, and the visible light channel camera can capture the visible light reflection of the object. Therefore, the dual-spectrum camera can simultaneously perform thermal imaging and optical imaging on the target. The temperature of the object can be measured according to the thermal imaging of the object.

[0064] In other embodiments of the present application, the sound collecting device and the visual collecting device can be integrated into the same device, and the specific forms of the two collecting devices are not specially limited in the present application.

[0065] The sound collecting device and the visual collecting device are deployed in the monitoring area, and the number of the sound collecting device and the visual collecting device deployed in the monitoring area can be determined according to the size of the monitoring area.

[0066] The positioning and tracking device is used for tracking the position information of the animal. For example, the positioning and tracking device can be a positioning collar, which can be worn on the neck of the animal to record the movement trajectory of the animal in real time. In an example, the positioning and tracking device can use the Beidou navigation positioning system to track the position (latitude and longitude information), movement speed, acceleration and other information of the animal.

[0067] In an example embodiment, each collecting device converts the collected signals into digital signals and transmits the digital signals to a device in which a multi-modal fusion engine is deployed through a network (such as the Internet of Things). The multi-modal fusion engine further extracts corresponding acoustic features, image features and positioning features from the digital sound signals, the digital signals corresponding to the images or videos and the position information, respectively.

[0068] Further, the multi-modal fusion engine is used to fuse the features collected by each collecting device according to the current scene to obtain multi-modal fusion features, and input the multi-modal fusion features into a risk decision module to identify whether there is an abnormal risk in the current monitoring environment.

[0069] If the risk decision module identifies that the multi-modal fusion features are consistent, the credit system is triggered to normally lend; if the risk decision module identifies that there is a conflict between the multi-modal fusion features, the conflict rate is greater than or equal to 15% and less than 30%, and the unmanned aerial vehicle is started for rechecking; if the risk decision module identifies that the conflict rate of the multi-modal fusion feature keys is greater than 30%, the credit limit is frozen.

[0070] Please refer to Figure 2 is a flowchart of a living body collateral state monitoring method provided by an embodiment of the present application, and the method is suitable forFigure 1 As shown in the multi-modal fusion engine and risk decision module, Figure 2 As shown in the method can comprise the following steps:

[0071] S101, acquiring the acoustic features, image features and positioning features of the monitoring object in the current monitoring area.

[0072] For the sound signal collected by the sound collection device in the monitoring area, the acoustic features of the animal can be extracted from the sound signal by using the model, for example, the improved ResNet-34 feature extraction network can be used to extract the acoustic feature vector (such as 128-dimensional vector) from the sound signal, and the acoustic feature vector includes the instantaneous frequency fluctuation feature, which can reflect the abnormal sound such as coughing.

[0073] Among them, the improved ResNet-34 feature extraction network introduces the multi-scale attention (Efficient Multi-Scale Attention, EMSA) mechanism, enhances the perception ability of the model to different scale features, and improves the robustness of acoustic feature extraction. In addition, according to the characteristics of the voiceprint recognition task, the residual module design is adjusted, for example, the convolution kernel parameters are adjusted or a more complex branch structure is introduced to adapt to the frequency domain features of the audio signal.

[0074] For the image or video collected by the visual collection device, the face key points (eye corners, nostrils, mouth corners, etc.) of the animal can be detected from the image or video by using the model (such as YOLOv8n), and the image features of the animal can be generated by combining 3D pose estimation. In addition, the image features include face orientation, body angle and other information.

[0075] For the position information (such as latitude and longitude, motion speed, acceleration, etc.) collected by the positioning tracking device, these information is converted into multi-dimensional space features, and the space features can include the distance from the fence, the motion trajectory curvature, etc.

[0076] S102, determining the target weight distribution matched with the current recognition scene based on the light intensity and environmental noise intensity of the environment where the monitoring object is located.

[0077] The light intensity, noise intensity and other factors of the same environment in different periods are different, resulting in different scenes of the environment. The acoustic features and image features corresponding to the same monitoring object collected are used to identify the scene of the environment where the monitoring object is currently located, such as strong light in the daytime, weak light at night, noisy environment and regular scene, etc. Further, different weight distributions of each modal feature are selected according to different scenes of the environment, so that the complementary effect of different modal features plays a role in the recognition process, and the weight of the feature affected by the environment scene is improved, for example, the weight coefficient of the acoustic feature is improved in the weak light environment at night.

[0078] In an example embodiment, the light intensity of the current environment can be identified according to the image features, and the scene of the current environment is determined, for example, the gray value (or brightness value) of the image reflects the brightness of each region of the image, that is, the light intensity, the higher the value of the gray or brightness, the stronger the light; otherwise, the lower the value of the gray or brightness, the weaker the light. Each color channel (such as RGB) of the color image can also reflect the light intensity difference in a similar manner. If the light intensity of the image or video is greater than a first preset light intensity threshold, it is determined that the current environment is a daytime strong light scene, and if the light intensity of the image or video is less than a second preset light intensity threshold, it is determined that the current environment is a night weak light scene. The first preset light intensity threshold is greater than the second preset light intensity threshold.

[0079] In another example embodiment, the noise intensity of the current environment can be determined according to the collected acoustic features, for example, the sound signal of other frequencies different from the fundamental frequency of the animal (such as the noise of the fan in the breeding farm and the call of other animals) can be identified from the acoustic feature signal, that is, the noise signal in the environment is identified, and the noise intensity is determined. If the noise intensity is greater than a preset threshold, it is determined that the current environment is a noisy environment.

[0080] If the current environment is not identified as a noisy environment based on the acoustic features of the monitored object, and the current environment is not identified as a night weak light scene or a daytime strong light scene based on the image features of the monitored object, it is determined that the current environment is a regular scene.

[0081] In one scenario, the current environment is identified as a daytime strong light scene, and the weight coefficient of the image features can be set to be greater than or equal to a first preset value, such as 50%. The sum of the weight coefficients of the acoustic features and the positioning features is (1-first preset value), and the weight coefficients of the acoustic features and the positioning features can be set according to actual needs, for example, the weight coefficients of the acoustic features and the positioning features are both 25%, which is not specially limited by the present application.

[0082] The effective degree of the image features is higher in the daytime strong light scene, so the weight coefficient of the image features is the largest, which can improve the accuracy of the identity recognition of the monitored object.

[0083] In another scenario, the current environment is identified as a night weak light scene, and the weight coefficient of the acoustic features can be greater than or equal to a second preset value, such as 60%. The sum of the weight coefficients of the image features and the positioning features is 40%. The specific weight coefficients of the image features and the positioning features can be set according to actual needs, which is not specially limited by the present application.

[0084] In a night weak light scene, the imaging signal-to-noise ratio of the camera of the visible light channel is low (for example, less than 10 dB), and the failure rate of face key point detection based on the image features of this scene is high, for example, 15%, and the missing recognition rate rises (for example, to 12%), which easily leads to the problem of live collateral being transferred. Therefore, in order to improve the recognition success rate in a night weak light environment, the application increases the weight coefficient of the acoustic feature in the fusion feature, that is, increases the proportion of the acoustic feature, and the effectiveness of the acoustic feature is higher, thereby improving the accuracy of identity recognition of the monitored object.

[0085] In another scenario, it is identified that the current environment is a noisy environment, and the weight coefficient of the acoustic feature is greater than or equal to a third preset value, for example, the third preset value is 70%. The sum of the weight coefficients of the image feature and the positioning feature is 30%.

[0086] In a farm, the fan and other animal sounds cause the environmental noise to be large, for example, greater than 70 dB, thereby causing the voiceprint collection signal-to-noise ratio to be small, for example, less than 15 dB, and finally causing the voiceprint recognition accuracy to decrease. Therefore, in order to improve the voiceprint recognition accuracy in a noisy environment, the proportion of the acoustic feature is increased, thereby improving the content of the acoustic feature information, and finally improving the voiceprint recognition accuracy.

[0087] S103, according to the target weight distribution, the acoustic feature, the image feature and the positioning feature of the monitored object are weighted and fused to obtain a multi-modal fusion feature.

[0088] In a scenario, after the weight distribution of each modal feature is determined, the features can be directly fused according to the weight coefficients of the modal features. For example, the weight coefficient of the image feature is 50%, the weight coefficient of the acoustic feature is 30%, and the weight coefficient of the positioning feature is 20%. Then, the features are multiplied by the corresponding weight coefficients and fused.

[0089] In another scenario, after the weight coefficient matching the current environment is determined, it is first judged whether the features of each modal are abnormal. If the features of any modal are abnormal, the weight coefficient of the abnormal feature is adjusted to a fourth preset value, for example, 70%, and the features of the three modal are fused according to the re-adjusted weight distribution. For example, the image feature is abnormal, the weight coefficient of the image feature is 70%, and the sum of the weight coefficients of the acoustic feature and the positioning feature is 30%. That is, the attention mechanism for abnormal modal features is increased, and the recognition accuracy and timeliness of the abnormal state are improved.

[0090] If the features of a single modal are not abnormal, the features of the three modal are fused according to the target weight distribution. The weighted fusion of the feature vectors is a linear combination of the feature vectors of different modal according to certain weights.

[0091] For example,Figure 3 As shown in FIG. 1, it is a schematic diagram of a multi-modal feature extraction and fusion process provided by an embodiment of the present application, which adopts a three-layer attention mechanism (i.e., intra-modal attention, inter-modal attention and state attention), automatically adjusts the weight coefficients of the features of each modality according to the current environment scene, and generates a unique fusion feature vector. In machine learning, the attention mechanism is a data processing method that simulates human cognitive characteristics. The core idea is to dynamically allocate weights so that the model can focus on key information and ignore irrelevant content when processing input.

[0092] As shown in FIG. 2, after collecting voiceprint, visual, and positioning signals of three different dimensions, the features in each dimension of information are extracted by the corresponding model. When extracting the features of each modality by the model, the intra-modal attention mechanism is used to obtain the features of the corresponding modality. For example, the intra-modal attention mechanism can strengthen the fundamental frequency features and formant features in the acoustic features. The intra-modal attention mechanism can strengthen the facial key point features in the image features. The intra-modal attention mechanism can strengthen the out-of-bound probability features in the positioning features. Figure 3 Then, in order to avoid the influence of factors such as light and noise on the feature recognition accuracy, the scene of the current monitoring environment is first judged, and then the weight coefficients of the features of each modality are dynamically adjusted according to the current scene. The inter-modal attention mechanism can be used to analyze the scene of the current monitoring environment based on the features of different modalities. For example, in the daytime strong light scene, the weight of the image features is greater than or equal to 50%; in the night weak light scene, the weight of the acoustic features is greater than or equal to 60%; in the noisy scene, the weight of the acoustic features is greater than or equal to 70%; and in the regular scene, the default weight is used for each modality feature.

[0093] Further, the state attention is used to dynamically identify whether there is an anomaly in each modality. If there is an anomaly in the features of any modality, the weight distribution of the features of each modality matched to the current scene is dynamically adjusted, for example, the weight coefficient of the modality feature with an anomaly is increased, such as in the daytime strong light scene, if the acoustic features have an anomaly, the weight coefficient of the acoustic features is increased to 70%, and the features of other modalities are correspondingly reduced. In this way, more attention can be paid to the modality features with an anomaly, so that the abnormal situation of the animal can be identified in time.

[0094] If the features of each modality do not have an anomaly, the feature fusion is performed according to the weight distribution corresponding to the scene, for example, in the daytime strong light scene, the weight coefficient of the image features is 50%, the weight coefficient of the acoustic features is 30%, and the weight coefficient of the positioning features is 20%.

[0095]

[0096] ​The feature fusion is performed according to the finally determined weight distribution of feature fusion, and a unique fusion feature vector is obtained, for example, a 134-dimensional fusion feature vector is finally fused.

[0097] In S104, the abnormal state of the monitored object is identified based on the multi-modal fusion feature.

[0098] In an exemplary embodiment, as shown in Figure 4 According to the multi-modal fusion feature, identity impersonation, health abnormality, location boundary crossing, and quantity abnormality can be identified.

[0099] In an exemplary embodiment, the process of identifying identity impersonation can include: if the fusion similarity of the multi-modal fusion feature of the monitored object is less than a first similarity threshold, it is determined that the monitored object is not the target object. The fusion similarity can be the similarity of the matching of the voiceprint and image feature in the multi-modal fusion feature with the voiceprint and image feature of the same animal. The first similarity threshold can be 80%, that is, when the fusion similarity of the voiceprint and vision is less than 80%, it is determined that the current monitored object has an abnormal situation of identity impersonation, that is, the identity of the collateral object is impersonated by other animals.

[0100] In another exemplary embodiment, the process of identifying the health abnormality of the monitored object can include: if the abnormal frequency band energy proportion of the acoustic feature in the multi-modal fusion feature of the monitored object is greater than a third threshold, and the eye temperature of the monitored object is greater than a fourth threshold based on the image feature in the multi-modal fusion feature, it is determined that the health state of the monitored object is abnormal.

[0101] For example, when the abnormal frequency band energy proportion of the acoustic feature in the multi-modal fusion feature is greater than 15%, it is identified that the animal has a coughing sound, and the infrared thermal imaging eye temperature is greater than 39°, indicating that the animal has a health abnormality. In order to improve the identification accuracy, if multiple coughing sounds are identified within a certain time, and the body temperature is greater than a set value, it is determined that there is a health abnormality, for example, if the animal has more than 5 coughing sounds per hour, and the body temperature is greater than 39°, it is determined that the animal has a health abnormality.

[0102] In an exemplary embodiment, the process of identifying the location boundary crossing of the monitored object can include: based on the positioning feature in the multi-modal fusion feature, it is identified that the monitored object exceeds the preset range of the electronic fence, and the crossing duration of the monitored object is recorded; if the crossing duration is greater than a first preset duration, it is determined that the monitored object has a location boundary crossing abnormality.

[0103] For example, if the positioning coordinates of a certain collateral object exceed the electronic fence, and the acoustic feature within the electronic fence cannot identify the object, it is determined that the object has a location boundary crossing abnormality. Further, the crossing duration of the monitored object is greater than a set duration, such as 120 minutes.

[0104] In an example embodiment, the number of mortgaged objects in the monitoring area is identified based on the multi-modal fusion features, and the process can include:

[0105] Based on the image features or acoustic features in the multi-modal fusion features corresponding to all monitoring objects in the current monitoring area, the number of all monitoring objects in the current monitoring area is identified as an identified number; if the deviation between the identified number and the account number is greater than a preset value, it is determined that the monitoring objects in the current monitoring area have a number anomaly. For example, the number of monitoring objects is counted based on the image features in the entire monitoring area to obtain a visual number, and the number of monitoring objects is counted based on the acoustic features in the entire monitoring area to obtain a voiceprint number, then the visual number and the voiceprint number are compared with the account number, if the visual number and / or the voiceprint number deviates from the account number and the deviation is greater than a certain value, such as 10%, it is determined that the monitoring objects in the monitoring area have a number anomaly.

[0106] In the above, the abnormal index system of each type of abnormal state can be as shown in Table 1:

[0107] Table 1

[0108]

[0109]

[0110] In addition, the living mortgaged object state monitoring method provided by the present application can also trigger different early warning responses according to the abnormal level, for example, including first-level early warning, second-level early warning and third-level early warning. The first-level early warning triggers a manual review process, such as a ranch personnel reviewing the condition of the mortgaged animal; the second-level early warning involves the financial lending institution, such as the financial lending institution starting an unmanned aerial vehicle to review the state of the animal in the ranch; the third-level early warning freezes the credit limit, and at the same time verifies the consistency of the multi-modal data through the smart contract and stores it on the chain, that is, stores the multi-modal data to the blockchain. The blockchain is deployed on the nodes of the financial institution, the ranch, etc., and is used to store the multi-modal feature hash value, the early warning record and other tamper-proof data.

[0111] In an example embodiment, according to Figure 4 After the four types of early warnings of identity early warning, health early warning, location early warning and number early warning are obtained as shown in Figure 5 Figure 4 The four types of early warnings identified are judged for early warning level as shown in Figure 5 ​G1-G4 in the above four early warnings, if the current early warning is a single modal feature caused by abnormality, it is determined as a first-level early warning, for example, position early warning and quantity early warning both belong to single modal feature abnormality, of course, it can also be other early warnings, and the present application does not specially limit this, this kind of scene can trigger the ranch review process; if the current early warning is an abnormality caused by two modal features, it is determined as a second-level early warning, for example, health early warning and identity early warning, or other early warning conditions are not specially limited, in this kind of scene, the financial lending institution can be intervened, such as starting the unmanned aerial vehicle review; if the current early warning is an abnormality caused by three modal features, it is determined as a third-level early warning, for example, identity early warning and quantity early warning occur at the same time, or other early warning conditions are not specially limited, in this kind of scene, the lending limit can be frozen.

[0112] In another exemplary embodiment, the early warning level corresponding to the abnormal state can be dynamically evaluated by calculating the conflict rate of the multi-modal feature confidence, for example, the conflict rate (conflict_rate) can be calculated according to the following formula:

[0113] conflict_rate = (1-(audio_score*visual_score*(1-gps_dev)))*100 (1)

[0114] Wherein, audio_score in formula 1 represents the confidence of acoustic feature, visual_score represents the confidence of image feature, and gps_dev represents the position boundary probability, (1-gps_dev) represents the confidence of being in the electronic fence.

[0115] If the conflict rate is less than or equal to a first threshold value (such as 15%), it is determined that the state of the monitoring object in the current monitoring area is normal;

[0116] If the conflict rate is greater than the first threshold value and less than a second threshold value (such as 15% < conflict rate < 20%), it is determined that the state of the monitoring object is abnormal and the abnormal state corresponds to a first-level early warning, and a review process is started, such as starting the unmanned aerial vehicle to review;

[0117] If the conflict rate is greater than or equal to the second threshold value and less than a third threshold value (such as 20% ≤ conflict rate < 30%), it is determined that the abnormal state corresponds to a second-level early warning, and a financial lending institution intervention process is started;

[0118] If the conflict rate is greater than or equal to the third threshold value (such as ≥ 30%), the credit limit is frozen and the multi-modal fusion data chain storage process is triggered.

[0119] The living collateral state monitoring method provided by the embodiment can obtain acoustic features, image features and positioning features of a monitoring object, identify a scene in which the monitoring object is currently located and determine a target weight distribution matched with the scene. The obtained acoustic features, image features and positioning features of the monitoring object are weighted and fused based on the target weight distribution to obtain multi-modal fusion features. Finally, the multi-modal fusion features are used to identify an abnormal state of the monitoring object, such as identity fraud, health abnormality, location out-of-bound, quantity abnormality and the like. The acoustic features can reflect voiceprint characteristics of an animal, the image features reflect visual characteristics of the animal, and the positioning features reflect location information of the animal. The complementary effects of multi-dimensional features are used to improve the identification accuracy and reduce the misidentification rate. Moreover, the scheme can identify a scene in the current environment and automatically adjust the weight distribution of each modal feature to generate a unique fusion feature vector. In addition, the robustness of the model to the environment is improved. For example, in a noisy and weak-light night scene, the multi-modal fusion features can improve the identification accuracy by 32% compared with single-dimensional features.

[0120] Further, the scheme can detect respiratory diseases of an animal in advance through voiceprint fundamental frequency microfluctuation and infrared thermal imaging, thereby reducing treatment costs.

[0121] Moreover, the method uses multi-level early warning responses to trigger corresponding early warning responses according to the abnormality level. When the abnormality level is high, the consistency of the multi-modal fusion data is verified by a smart contract and stored on a chain. The multi-modal data are mutually verified, and the voiceprint, visual and positioning features form a complete evidence chain. Any tampering with single-modal data will cause the multi-modal fusion data consistency verification to fail, such as modifying the positioning data but the voiceprint and visual features not matching.

[0122] The above describes a living collateral state monitoring method provided by an embodiment of the application. The following describes a device for executing the living collateral state monitoring method.

[0123] Please refer to Figure 6 , Figure 6 FIG. 1 is a structural schematic diagram of a living collateral state monitoring device provided by an embodiment of the application. As shown in the figure, the device includes: Figure 6

[0124] A feature acquisition module 101 is configured to acquire acoustic features, image features and positioning features of a monitoring object in a current monitoring area.

[0125] A weight determination module 102 is configured to determine a target weight distribution matched with a current identification scene based on an illumination intensity and an environmental noise intensity of an environment in which the monitoring object is located.

[0126] In a possible implementation manner, the weight determination module 102 is specifically configured to: ​

[0127] According to the light intensity of the environment where the monitoring object is located, it is identified that the current environment scene is daytime strong light, and the weight coefficient of the image feature is greater than or equal to a first preset value, the first preset value is greater than or equal to the sum of the weight coefficients of the acoustic feature and the positioning feature;

[0128] According to the light intensity of the environment where the monitoring object is located, it is identified that the current environment scene is night weak light, and the weight coefficient of the acoustic feature is greater than or equal to a second preset value, the second preset value is greater than the sum of the weight coefficients of the image feature and the positioning feature;

[0129] According to the environmental noise intensity of the environment where the monitoring object is located, it is identified that the current environment scene is a noisy environment, and the weight coefficient of the acoustic feature is greater than or equal to a third preset value, the third preset value is greater than the sum of the weight coefficients of the image feature and the positioning feature.

[0130] The feature fusion module 103 is configured to fuse the acoustic feature, the image feature and the positioning feature of the monitoring object according to the target weight distribution to obtain a multi-modal fusion feature.

[0131] In a possible implementation, the feature fusion module 103 is specifically configured to:

[0132] determine whether there is an anomaly based on the feature of a single modality of the monitoring object;

[0133] if the feature of any modality is abnormal, the weight of the abnormal feature is a fourth preset value, the fourth preset value is greater than the sum of the weights of the features without anomaly, and the feature fusion is performed on the features of the three modalities according to the re-adjusted weight distribution;

[0134] if the features of the single modalities are all normal, the feature fusion is performed on the features of the three modalities according to the target weight distribution.

[0135] The state recognition module 104 is configured to recognize the abnormal state of the monitoring object based on the multi-modal fusion feature.

[0136] In a possible implementation, the state recognition module 104 is specifically configured to:

[0137] obtain the similarity between the feature of any modality in the multi-modal fusion feature of the monitoring object and the feature of the corresponding modality of the target object, obtain the voiceprint confidence, the image confidence and the confidence of being in the electronic fence;

[0138] determine the conflict rate of the multi-modal feature based on the voiceprint confidence, the image confidence and the confidence of being in the electronic fence corresponding to the monitoring object;

[0139] If the conflict rate is less than or equal to a first threshold value, it is determined that the monitoring object state of the current monitoring area is normal;

[0140] If the conflict rate is greater than the first threshold value and less than a second threshold value, a review process is triggered;

[0141] If the conflict rate is greater than or equal to the second threshold value, the credit limit is frozen and a multi-modal fusion data on-chain storage process is triggered.

[0142] In another possible implementation, the state identification module 104 is specifically configured to:

[0143] If the fusion similarity of the multi-modal fusion features of the monitoring object is less than a first similarity threshold value, it is determined that the monitoring object is not a target object;

[0144] If the abnormal frequency band energy proportion of the acoustic feature in the multi-modal fusion features of the monitoring object is greater than a third threshold value, and the eye temperature of the monitoring object is greater than a fourth threshold value based on the image feature in the multi-modal fusion features, it is determined that the health state of the monitoring object is abnormal.

[0145] Based on the positioning feature in the multi-modal fusion features, it is identified that the monitoring object exceeds a preset range of an electronic fence, and the time length of the monitoring object crossing the boundary is recorded; if the time length of crossing the boundary is greater than a first preset time length, it is determined that the monitoring object has a location boundary crossing anomaly.

[0146] Based on the image feature or the acoustic feature in the multi-modal fusion features corresponding to all monitoring objects in the current monitoring area, the number of all monitoring objects in the current monitoring area is identified to obtain an identified number; if the deviation between the identified number and the account number is greater than a preset value, it is determined that the monitoring objects in the current monitoring area have a quantity anomaly.

[0147] Further, the device can further include a pre-warning module, which is specifically configured to:

[0148] determine a pre-warning level corresponding to the current abnormal state, and if the current abnormal state is caused by a single modal feature anomaly, it is determined as a first-level pre-warning, and a review process is started;

[0149] if the current abnormal state is caused by two modal feature anomalies, it is determined as a second-level pre-warning, and a financial lending institution intervention process is started;

[0150] if the current abnormal state is caused by three modal feature anomalies, it is determined as a third-level pre-warning, the credit limit is frozen, and a multi-modal fusion data on-chain storage process is triggered.

[0151] The living collateral state monitoring device provided by the embodiment acquires acoustic features, image features and positioning features of a monitoring object, identifies a scene in which the monitoring object is currently located and determines a target weight distribution matched with the scene. The obtained acoustic features, image features and positioning features of the monitoring object are weighted and fused based on the target weight distribution to obtain multi-modal fusion features. Finally, the abnormal state of the monitoring object, such as identity fraud, health abnormality, location out-of-bound, quantity abnormality and the like, is identified based on the multi-modal fusion features. The acoustic features can reflect the voiceprint characteristics of the animal, the image features reflect the visual characteristics of the animal, and the positioning features reflect the location information of the animal. The complementary effects of the features in multiple dimensions are utilized to improve the identification accuracy and reduce the misidentification rate. Moreover, the scheme can identify the scene of the current environment and automatically adjust the weight distribution of each modal feature according to the scene characteristics to generate a unique fusion feature vector, and the robustness of the model to the environment is improved.

[0152] An electronic device is also provided in the embodiments of the present application. Referring to Figure 7 Fig. 1 shows a structural schematic diagram of an electronic device suitable for implementing the electronic device in the embodiments of the present application. The electronic device in the embodiments of the present application can include, but is not limited to, terminal devices such as notebook computers, PDAs (personal digital assistants), PADs (tablet computers), desktop computers and the like, or can also be cloud devices such as cloud servers and the like. Figure 7 The electronic device shown is merely an example and should not impose any limitation on the functions and use range of the embodiments of the present application.

[0153] As Figure 7 shown, the electronic device can include a processing device (such as a central processing unit, a graphics processing unit, etc.) 201, which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 202 or loaded from a storage device 208 into a random access memory (RAM) 203. In a state where the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 203. The processing device 201, the ROM 202 and the RAM 203 are connected to each other through a bus 204. An input / output (I / O) interface 205 is also connected to the bus 204.

[0154] Generally, the following devices can be connected to the I / O interface 205: input devices 206 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope and the like; output devices 207 including, for example, a liquid crystal display (LCD), a speaker, a vibrator and the like; storage devices 208 including, for example, a memory card, a hard disk and the like; and communication devices 209. The communication devices 209 can allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 7Electronic devices having various apparatuses are shown, but it is understood that all of the illustrated apparatuses are not required, and that more or less than the illustrated apparatuses can be implemented.

[0155] The embodiment of the present application further provides a computer program product comprising computer readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the living collateral state monitoring methods provided by the embodiments of the present application.

[0156] The embodiment of the present application further provides a computer readable storage medium, which carries one or more computer programs, and when the one or more computer programs are executed by an electronic device, the electronic device can implement any of the living collateral state monitoring methods provided by the embodiments of the present application.

[0157] In addition, it should be noted that the apparatus embodiments described above are merely illustrative, and the units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, i.e., they can be located in one place or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment according to actual needs. In addition, in the apparatus embodiment provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which can be implemented as one or more communication buses or signal lines.

[0158] From the above description of the embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software and necessary general hardware, and of course can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. Generally, functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure for implementing the same function can also be various, such as analog circuit, digital circuit or special circuit, etc. However, for the present application, software program implementation is a better embodiment. Based on this understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer's floppy disk, U disk, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc., including a plurality of instructions to make a computer device (which can be a personal computer, training device, or network device, etc.) execute the methods described in various embodiments of the present application.

[0159] In the above embodiments, all or part can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in the form of a computer program product in whole or in part.

[0160] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transmitted from one website, computer, training device or data center to another website, computer, training device or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. integrated with one or more available media sets. The available media can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk (SSD)), etc.

Claims

1. A method of monitoring the condition of a living collateral, characterized by, The method comprises: acquiring acoustic features, image features and positioning features of a monitoring object in a current monitoring area; determining a target weight distribution matched with a current identification scene based on illumination intensity and ambient noise intensity of an environment where the monitoring object is located; weighting and fusing the acoustic features, image features and positioning features of the monitoring object according to the target weight distribution to obtain multi-modal fusion features; identifying an abnormal state of the monitoring object based on the multi-modal fusion features.

2. The method of claim 1, wherein, The method of determining the target weight distribution matched with the current identification scene based on the illumination intensity and ambient noise intensity of the environment where the monitoring object is located comprises: identifying that the current environment scene is daytime strong light according to the illumination intensity of the environment where the monitoring object is located, and then setting a weight coefficient of the image features to be greater than or equal to a first preset value, which is greater than or equal to a sum of weight coefficients of the acoustic features and the positioning features; identifying that the current environment scene is nighttime weak light according to the illumination intensity of the environment where the monitoring object is located, and then setting a weight coefficient of the acoustic features to be greater than or equal to a second preset value, which is greater than a sum of weight coefficients of the image features and the positioning features; identifying that the current environment scene is a noisy environment according to the ambient noise intensity of the environment where the monitoring object is located, and then setting a weight coefficient of the acoustic features to be greater than or equal to a third preset value, which is greater than a sum of weight coefficients of the image features and the positioning features.

3. The method according to claim 1 or 2, characterized in that, The method of weighting and fusing the acoustic features, image features and positioning features of the monitoring object according to the target weight distribution to obtain multi-modal fusion features comprises: determining whether there is an anomaly based on features of a single modality of the monitoring object; if there is an anomaly in any modality, setting a weight of the feature with the anomaly to be a fourth preset value, which is greater than a sum of weights of features without anomaly, and then fusing the features of the three modalities according to the weight distribution after adjustment; if there is no anomaly in the features of the single modality, fusing the features of the three modalities according to the target weight distribution.

4. The method of claim 1, wherein, The method of identifying an abnormal state of the monitoring object based on the multi-modal fusion features comprises: if a fusion similarity of the multi-modal fusion features of the monitoring object is less than a first similarity threshold, determining that the monitoring object is not a target object; if an abnormal frequency band energy ratio of the acoustic features in the multi-modal fusion features of the monitoring object is greater than a third threshold, and an eye temperature of the monitoring object determined based on the image features in the multi-modal fusion features is greater than a fourth threshold, determining that a health state of the monitoring object is abnormal.

5. The method according to claim 1 or 4, characterized in that, The method of identifying an abnormal state of the monitoring object based on the multi-modal fusion features comprises: identifying that the monitoring object exceeds a preset range of an electronic fence based on the positioning features in the multi-modal fusion features, and recording a time length of the monitoring object when crossing the boundary; if the time length of crossing the boundary is greater than a first preset time length, determining that the monitoring object has a location crossing boundary anomaly.

6. The method according to claim 1 or 4, characterized in that, The method of identifying an abnormal state of the monitoring object based on the multi-modal fusion features comprises: The number of all monitoring objects in the current monitoring area is identified based on image features or acoustic features in the multi-modal fusion features corresponding to all monitoring objects in the current monitoring area, to obtain an identified number; If a deviation between the identified number and the account number is greater than a preset value, it is determined that the number of monitoring objects in the current monitoring area is abnormal.

7. The method of claim 1, wherein, After the abnormal state of the monitoring object is identified based on the multi-modal fusion features, the method further includes: determining a warning level corresponding to the current abnormal state, if the current abnormal state is caused by a single modal feature abnormality, determining it as a first-level warning, and starting a review process; if the current abnormal state is caused by two modal feature abnormalities, determining it as a second-level warning, and starting a financial lending institution intervention process; if the current abnormal state is caused by three modal feature abnormalities, determining it as a third-level warning, freezing the credit limit, and triggering a multi-modal fusion data chain storage process.

8. The method of claim 1, wherein, After the abnormal state of the monitoring object is identified based on the multi-modal fusion features, the method further includes: obtaining a voiceprint confidence, an image confidence, and a confidence of being in an electronic fence in the multi-modal fusion features of the monitoring object; determining a conflict rate of the multi-modal features based on the voiceprint confidence, the image confidence, and the confidence of being in the electronic fence corresponding to the monitoring object; if the conflict rate is greater than the first threshold value and less than the second threshold value, determining that the abnormal state corresponds to a first-level warning and starting a review process; if the conflict rate is greater than or equal to the second threshold value and less than the third threshold value, determining that the abnormal state corresponds to a second-level warning and starting a financial lending institution intervention process; if the conflict rate is greater than or equal to the third threshold value, determining that the abnormal state corresponds to a third-level warning, freezing the credit limit, and triggering a multi-modal fusion data chain storage process.

9. A living collateral condition monitoring apparatus, characterized by, It includes: a feature acquisition module for acquiring acoustic features, image features, and positioning features of monitoring objects in a current monitoring area; a weight determination module for determining a target weight distribution matched with a current identification scene based on the light intensity and environmental noise intensity of the environment where the monitoring object is located; a feature fusion module for weighting and fusing the acoustic features, image features, and positioning features of the monitoring object according to the target weight distribution, to obtain multi-modal fusion features; a state identification module for identifying the abnormal state of the monitoring object based on the multi-modal fusion features.

10. An electronic device, comprising: It includes at least one processor and a memory connected to the processor, wherein: the memory is used to store a computer program; the processor is used to execute the computer program, so that the electronic device can implement the living collateral state monitoring method according to any one of claims 1 to 8.

Images