Method for determining risk score, storage medium, electronic device and product

Abstract

Embodiments of the present application provide a risk score determination method, a storage medium, an electronic device and a product, related to the field of communication, based on fiber to the room technology, comprising: determining a plurality of first probability values corresponding to a plurality of preset risk type events respectively according to image data of a target area, and determining a plurality of second probability values corresponding to the plurality of preset risk type events respectively according to audio data of the target area; determining a risk score of the target area according to the plurality of first probability values corresponding to the plurality of preset risk type events respectively and the plurality of second probability values corresponding to the plurality of preset risk type events respectively; and determining a target risk score of the target area according to a score correction factor and the risk score, solving the problem that risk assessment in related technologies relies on a single modal perception technology, leading to inaccurate risk assessment, and thereby improving the accuracy of risk assessment in a complex home environment.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of communications, and more specifically, to a method for determining risk scores, a storage medium, an electronic device, and a product. Background Technology

[0002] In existing technologies, smart home monitoring systems generally rely on single-modal sensing technologies to achieve risk identification, such as visual monitoring, voice wake-up, or wearable sensing. However, single-modal sensing technologies have limited information dimensions, making it difficult to accurately identify risks in complex scenarios.

[0003] Optionally, vision-based monitoring systems are prone to failure due to furniture obstruction, dim lighting, limited field of view, or distance from the person. They are particularly incapable of detecting sudden illnesses without obvious movement (such as stroke or myocardial infarction) or silent suffocation. Furthermore, their algorithms often misjudge everyday behaviors (such as bending over to pick up an object or resting in a reclining chair) as falls, leading to frequent false alarms. Voice-interactive systems generally rely on keyword matching mechanisms, making it difficult to distinguish between genuine calls for help and background sounds such as television programs, radio broadcasts, or phone calls. Moreover, the recognition rate drops sharply under conditions of multi-person conversations, music playback, or environmental noise interference. Although wearable devices can monitor physiological or motor parameters, their actual usage rate is low because they rely on users to actively wear them. They can only detect abnormal limb movements and cannot identify non-contact risks such as fires, gas leaks, or illegal intrusions.

[0004] Therefore, in the existing technology, risk assessment relies on single-modal sensing technology, which leads to inaccurate risk assessment, and no effective solution has yet been proposed. Summary of the Invention

[0005] This invention provides a method for determining risk scores, a storage medium, an electronic device, and a product to at least address the problem in related technologies where risk assessment relies on a single-modal sensing technology, leading to inaccurate risk assessments.

[0006] According to an embodiment of the present invention, a method for determining a risk score is provided, comprising: determining first probability values ​​corresponding to a plurality of preset risk type events based on image data of a target area, and determining second probability values ​​corresponding to a plurality of preset risk type events based on audio data of the target area; determining a risk score for the target area based on the first probability values ​​and the second probability values ​​corresponding to the plurality of preset risk type events; and determining a target risk score for the target area based on a score correction factor and the risk score, wherein the score correction factor is used to indicate the influence of time and / or the scene type of the target area on the risk score.

[0007] According to another embodiment of the present invention, a risk score determination apparatus is provided, comprising: a first determination module, configured to determine first probability values ​​corresponding to multiple preset risk type events based on image data of a target area, and to determine second probability values ​​corresponding to multiple preset risk type events based on audio data of the target area; a second determination module, configured to determine a risk score of the target area based on the first probability values ​​and the second probability values ​​corresponding to the multiple preset risk type events; and a third determination module, configured to determine a target risk score of the target area based on a score correction factor and the risk score, wherein the score correction factor is used to indicate the influence of time and / or the scene type of the target area on the risk score.

[0008] According to yet another embodiment of the present invention, a computer-readable storage medium is also provided, wherein a computer program is stored therein, wherein the computer program is configured to perform the steps in any of the above method embodiments when executed.

[0009] According to yet another embodiment of the present invention, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0010] According to yet another embodiment of the present invention, a computer program product is also provided, comprising a computer program that, when executed by a processor, implements the steps in any of the above method embodiments.

[0011] Through the above embodiments of the present invention, based on image data and audio data, the visual and audio probability values ​​corresponding to multiple preset risk type events are determined respectively. The visual and audio probability values ​​are then fused to determine the risk score of the target area. The risk score is then corrected based on a score correction factor to obtain the target risk score. Therefore, this solves the problem of inaccurate risk assessment caused by reliance on single-modal perception technology in related technologies, thereby improving the accuracy of risk assessment in complex home environments. Attached Figure Description

[0012] Figure 1 This is a hardware structure block diagram of a computer terminal operating according to an embodiment of the present invention;

[0013] Figure 2 This is a flowchart of a risk scoring method according to an embodiment of the present invention;

[0014] Figure 3 This is a schematic diagram of the architecture of a risk scoring method according to an embodiment of the present invention;

[0015] Figure 4 This is a schematic diagram of the visual modality analysis module of a risk scoring determination method according to an embodiment of the present invention;

[0016] Figure 5 This is a schematic diagram of a speech modality parsing module in a risk scoring determination method according to an embodiment of the present invention;

[0017] Figure 6 This is a structural block diagram of a risk scoring determination device according to an embodiment of the present invention. Detailed Implementation

[0018] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings and examples.

[0019] It should be noted that the terms "first," "second," etc., in the specification, claims, and drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0020] The methods and embodiments provided in this application can be executed on a mobile terminal, a computer terminal, or a similar computing device. Taking running on a computer terminal as an example, Figure 1 This is a hardware structure block diagram of the computer terminal used in the embodiments of the method of the present invention. For example... Figure 1 As shown, a computer terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor (MCU) or a field-programmable gate array (FPGA)) and a memory 104 for storing data are also shown. The computer terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the computer terminal described above. For example, the computer terminal may also include components that are more complex than those described above. Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0021] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to a risk scoring determination method in this embodiment of the invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thereby implementing the aforementioned method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to a computer terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0022] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider for the computer terminal. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0023] This embodiment provides a method for determining a risk score that runs on the aforementioned computer terminal. Figure 2 This is a flowchart of a risk scoring method according to an embodiment of the present invention, such as... Figure 2 As shown, the process includes the following steps:

[0024] Step S202: Determine the first probability value corresponding to multiple preset risk type events based on the image data of the target area, and determine the second probability value corresponding to multiple preset risk type events based on the audio data of the target area;

[0025] It should be noted that the aforementioned image and audio data can be transmitted using Fiber to the Room (FTTR) technology.

[0026] In an exemplary embodiment, determining the first probability values ​​corresponding to multiple preset risk type events based on image data of the target area can be achieved through the following steps S302-S306:

[0027] Step S302: Obtain the first confidence level of the risk target in the image data and the second confidence level of the abnormal posture of the target object in the image data, wherein the risk target is a biological individual or a non-biological individual, and the target object is a biological individual;

[0028] Among them, the first confidence level c v The first confidence level represents the detection confidence of whether a risky target (i.e., an individual that poses a potential risk to the target object) exists in the image, such as a "knife," "flame," "snake," or "wasp." Its value range is [0,1], and it indicates the computer vision (CV) model's certainty in determining the existence of the risky target. The second confidence level p... v This characterizes whether the target object (i.e., a person or animal) in the image exhibits an abnormal posture, such as "lying prone," "rolling over," or "irregular twitching," which are posture features that deviate significantly from normal behavior. It is output by the human keypoint detection model and also takes values ​​in [0,1], which are used to indicate the degree of significance of the abnormal posture of the target object by the computer vision model.

[0029] Step S304: Determine the visual risk score and visual risk uncertainty score of the target area based on the first confidence level and the second confidence level, wherein the visual risk score is used to indicate the probability of a risk event occurring in the target area determined based on the image data, and the visual risk uncertainty score is used to indicate the degree of unreliability of the visual risk score;

[0030] It should be noted that the higher the visual risk uncertainty score, the higher the unreliability of the visual risk score.

[0031] In an exemplary embodiment, determining a visual risk score and a visual risk uncertainty score for the target area based on a first confidence level and a second confidence level includes: determining the visual risk score based on the product of a first weight and the first confidence level, the product of a second weight and the second confidence level, and the product of a third weight and the first confidence level and the second confidence level, wherein the first weight is the weight corresponding to the first confidence level, the second weight is the weight corresponding to the second confidence level, and the third weight is the weight corresponding to the product of the first confidence level and the second confidence level; and determining the average value of the sum of the first confidence level and the second confidence level, and determining the visual risk uncertainty score based on the difference between a first value and the average value, wherein the first value is used to indicate the value corresponding to the highest reliability level of the visual risk score.

[0032] The visual risk score r is determined according to the following formula. v :r v =w1·cv +w2·p v +w3·(c v ·p v ), where w1+w2+w3=1, w1 is the weight corresponding to the first confidence level, w2 is the weight corresponding to the second confidence level, and w3 is the weight corresponding to the product of the first and second confidence levels; and the visual risk uncertainty score δ is determined according to the following formula. v δ v =1-(c v +p v ) / 2, where, when the first confidence level c v With the second confidence level p v When both values ​​approach 1, it indicates that the visual modality simultaneously possesses high-confidence detection results for risky targets and high-confidence recognition results for abnormal postures of target objects. At this point, the visual risk uncertainty score δ... v =1-(c v +p v A visual risk score that is close to 0 (c / 2) indicates a high degree of reliability; conversely, a score that is close to 0 (c / 2) indicates a lower degree of reliability. v or p v If any value is below 1, for example, only low-confidence target contours are detected (c v ≈0.3) and no valid human posture was identified (p v ≈0.1), then (c v +p v The value of ) / 2 is too low, resulting in δ v Increase, at this time, δ v The visual risk score has a low reliability because the perceptual evidence provided by the visual modality is partially missing, vague, or contradictory.

[0033] Step S306: Determine the first probability value corresponding to each of the plurality of preset risk type events based on the visual risk score and the visual risk uncertainty score.

[0034] In an exemplary embodiment, when the plurality of preset risk type events include risk events, risk-free events, and uncertain risk events, determining the first probability value corresponding to each of the plurality of preset risk type events based on the visual risk score and the visual risk uncertainty score includes: determining a first difference between a first value and the visual risk uncertainty score, and determining the first probability value corresponding to the risk event based on the product of the first difference and the visual risk score, wherein the first value is used to indicate the value corresponding to the highest reliability of the visual risk score; determining a second difference between the first value and the visual risk score, and determining the first probability value corresponding to the risk-free event based on the product of the first difference and the second difference; and determining the first probability value corresponding to the uncertain risk event based on the visual risk uncertainty score.

[0035] In this embodiment, the preset risk type events include: risk events (R), risk-free events (¬R), and uncertain risk events (Θ), where Θ = {R, ¬R} represents an uncertain state that cannot be clearly attributed at present. The visual risk score r output by the visual modality... v ∈[0,1] and visual risk uncertainty score δ v Based on the Basic Probability Assignment (BPA) principle of Dempster-Shafer (DS) evidence theory, the following deterministic mapping relationship is constructed for ∈[0,1]:

[0036] For the first probability value m corresponding to the risk event (R) v ({R}), determine the first value = 1, the first value is used to indicate the value corresponding to the highest reliability of the visual risk score. Calculate the first value and the visual risk uncertainty score δ. v First difference: (1-δ v The first difference represents the proportion of valid certainty in visual evidence, i.e., the weight of credible evidence that can be used to support risk judgment after eliminating uncertainty. The first difference is then compared with the visual risk score r. v Multiplying them together gives the first probability value m of the risk event R. v ({R})=r v ·(1-δ v ).

[0037] For the first probability value m corresponding to the risk-free event (¬R) v ({¬R}), calculate the first value and the visual risk score r v The second difference: (1-r v The second difference represents the non-risk confidence level that no risky behavior was observed. The second difference is compared with the first difference (1-δ).v Multiplying these two values ​​yields the first probability value m of the risk-free event ¬R. v ({¬R})=(1-r v )·(1-δ v ).

[0038] For the first probability value m corresponding to the uncertain risk event (Θ) v (Θ) will be used to determine the visual risk uncertainty score δ. v The first probability value m of this uncertain state v (Θ)=δ v The first probability value corresponding to an uncertain risk event represents the cognitive uncertainty of the visual modality caused by factors such as occlusion, low resolution, abnormal lighting, or incomplete target.

[0039] In an exemplary embodiment, determining the second probability values ​​corresponding to multiple preset risk type events based on the audio data of the target area can be achieved through the following steps S402-S406:

[0040] Step S402: Determine the sound source consistency score of the target area based on the audio data and the image data;

[0041] For example, based on the real-time audio signal collected by the audio sensor and combined with the spatial location information of the target object output by the visual modality, the sound source consistency score c is calculated. a ∈[0,1], where the sound source consistency score is used to measure the spatial consistency between audio events and visual targets in the spatial dimension.

[0042] In an exemplary embodiment, determining a sound source consistency score for a target region based on the audio data and the image data includes: performing target detection on the audio data to determine the auditory azimuth angle of a target object in the target region, and performing target detection on the image data to determine the visual azimuth angle of the target object in the target region; and determining the sound source consistency score based on the auditory azimuth angle and the visual azimuth angle.

[0043] Obtain the auditory azimuth θ of abnormal audio events (such as screams or cries for help) in three-dimensional space. a The target center point of the target object is determined from the image data. After coordinate system transformation, the visual azimuth angle θ of the corresponding target object in the same spatial coordinate system as the auditory perception is obtained. v Then, the azimuth deviation between the two is calculated as Δθ = |θ a -θ v | Determining the source consistency score c based on azimuth deviation a =max(0,1-|θ a -θv | / θ th ), where the sound source consistency score is used to indicate the consistency between the auditory direction and the visual target direction, θ th A preset angle threshold (e.g., 10°) is used to tolerate sensor errors and small target deviations. When Δθ ≤ θ th At that time, c a >0 indicates that the sound source and the visual target are located in the same physical region; when Δθ>θ th At that time, c a =0 indicates that the sound source and the visual target are not located in the same physical area, and may originate from non-target sound sources such as television or radio.

[0044] Optionally, target detection is performed on the audio data to determine the auditory azimuth of the target object in the target area, including: calculating the cross-correlation function of the sub-audio data based on the sub-audio data collected by each pair of audio acquisition devices, determining the time difference between the corresponding sub-audio data collected by each pair of audio acquisition devices based on the cross-correlation function of the sub-audio data, and determining the auditory azimuth of the sound source in the target area based on the distance between the two pairs of audio acquisition devices and the time difference, wherein the audio data is generated based on the sub-audio data.

[0045] In an exemplary embodiment, after performing target detection on the audio data to determine the auditory azimuth of a target object in the target region, and performing target detection on the image data to determine the visual azimuth of the target object in the target region, the method further includes: determining the azimuth difference between the auditory azimuth and the visual azimuth, and determining the audio-visual time difference between the audio data and the image data; and storing the audio data and the image data if the azimuth difference is less than an azimuth threshold and the audio-visual time difference is less than a time difference threshold.

[0046] In obtaining the visual azimuth angle θ v With auditory azimuth θ a Then, calculate the azimuth difference Δθ between the two: Δθ = |θ a -θ v The azimuth difference indicates the consistency between the auditory direction and the visual target direction; that is, the smaller the Δθ, the more likely the sound source and the visual target are located in the same physical area. Simultaneously, the timestamps of the audio frame and the corresponding video frame are compared to calculate the audio-visual time difference Δt: Δt = |t| a -t v |, where t a t is the timestamp of the audio event. v As the timestamp corresponding to the visual frame, the audio-visual time difference reflects the synchronization of audio data and image data in the time dimension.

[0047] When Δθ≤α (α is the azimuth threshold, such as 10°), it indicates that the sound source and the visual target belong to the same area in space, excluding interference from unrelated sound sources such as "screaming on TV" or "someone shouting outside the window"; when Δt≤t threshold (t) threshold When the time difference threshold is 20ms, it indicates that the sound and image are highly synchronized in time, eliminating false events caused by asynchronous device sampling or transmission delay.

[0048] When both of the above conditions are met, the event is determined to be a high-risk event, and a trusted evidence chain generation process is executed: audio and video data (including precise timestamps, device IDs, and sensor status) are captured from 3 seconds before to 5 seconds after the event, and high compression processing is performed using a lightweight H.265 and Opus encoder (video bitrate ≤10Mbps, audio ≤64kbps). The compressed evidence package is then synchronously written to a local secure storage module and / or a cloud server for storage.

[0049] If Δθ>10° or Δt>20ms, the data will not be written to the local secure storage module. Instead, the audio and video data will be uploaded to the cloud as a log to be verified, which will be used for manual review to avoid generating invalid evidence due to sensor misjudgment or environmental interference.

[0050] Step S404: Determine the target semantic information of the audio data based on the parsing result of the audio data, determine the risk level score of the target object in the target area based on the target semantic information, and normalize the decibel information of the audio data to determine the decibel score of the target area.

[0051] For example, the audio is divided into abnormal audio and normal speech by a speech classification model; abnormal audio is further classified (such as screaming, calling for help, glass breaking); normal speech is enhanced by an AI deep noise reduction module, and then converted into text by Automatic Speech Recognition (ASR) and input into a Large Language Model (LLM) for semantic understanding to obtain the target semantic information.

[0052] The target semantic information includes, but is not limited to: distress signals (e.g., I fell down, someone come quick, fire), semantic urgency (e.g., mild discomfort, emergency call for help), and whether the context supports the real event (excluding interference from movie clips, phone calls, etc.).

[0053] Risk level score is determined based on target semantic information. a ∈[0,1], the risk level score indicates the strength of the semantic danger implied by the speech content. For example, the risk level score corresponding to "help" is...a =0.95; the corresponding s for "I feel a little dizzy" a =0.6; s corresponding to no semantic content or non-dangerous semantics a =0. Simultaneously, the decibel information of the audio data is normalized to obtain a decibel score d. a ∈[0,1], the decibel score is used to quantify the physical intensity characteristics of sound.

[0054] Step S406: Determine the second probability value corresponding to each of the multiple preset risk type events based on the risk level score, the decibel score, and the sound source consistency score.

[0055] In an exemplary embodiment, determining the second probability values ​​corresponding to the plurality of preset risk type events based on the risk level score, the decibel score, and the sound source consistency score can be achieved through the following steps S502-S504:

[0056] Step S502: Determine the hearing risk score and hearing risk uncertainty score of the target area based on the risk level score, the decibel score, and the sound source consistency score;

[0057] It should be noted that the hearing risk uncertainty score is used to indicate the degree of unreliability of the hearing risk score.

[0058] In one exemplary embodiment, determining an auditory risk score and an auditory risk uncertainty score for the target region based on the risk level score, the decibel score, and the sound source consistency score includes: determining the auditory risk score based on the product of a fourth weight and the risk level score, the product of a fifth weight and the decibel score, the product of a sixth weight and the risk level score and the decibel score, and the product of a seventh weight and the sound source consistency score; and determining the auditory risk uncertainty score based on the average of the risk level score, the decibel score, and the sound source consistency score.

[0059] The hearing risk score r is determined using the following formula. a :r a =w4·s a +w5·d a +w6·(s a ·d a )+w7·c a Where w4+w5+w6+w7=1, w4·s a Used to indicate the intensity of the dangerous intent implied in the speech semantics, such as the high s-value corresponding to explicit distress signals like "help," "help me," and "I've fainted." a The value, while "I'm feeling a little unwell" or non-urgent semantics correspond to low s aValue; w4 is the fourth weight corresponding to the risk level score; w5·d a A supplementary criterion used to indicate the urgency of an event by the physical intensity of an audio signal, d a The normalized decibel score indicates the loudness of abnormal sounds. w5 is the fifth weight corresponding to the decibel score, used to enhance the response to sudden, non-verbal hazards (such as falls, impacts, and broken glass); w6·(s a ·d a This is used to determine the true risk level under the combined effect of the two. When the semantic intent is significant (s a High) and accompanied by high volume (d) a When the volume is high (s), it is highly likely to be a real emergency; conversely, if the meaning is clear but the volume is weak (s), it is likely a real emergency. a high, d a (Low), which may indicate that the trapped person is physically weak or restricted. Although the risk level is high, the urgency is low. w6 is the sixth weight; w7·c a Used to ensure consistency between voice events and visual targets, if c a =0 (sound source and target space are separated), then this item directly suppresses the risk score and prevents false events such as "TV playing a distress call" from being triggered.

[0060] Furthermore, the hearing risk uncertainty score δ is determined according to the following formula. a δ a =1-(s a +d a +c a ) / 3, when s a d a c a If all three are close to 1 (semantic clarity, high volume, and aligned sound sources), then their average value approaches 1, δ. a A value close to 0 indicates that the voice evidence is highly consistent and credible; when any indicator is low (such as s), it indicates that the voice evidence is highly consistent and credible. a =0.9 but c a =0 indicates "distress was heard but no one was seen"; then the average value decreases, δ a The rise indicates that there are uncertainties in the semantic and spatial aspects of the speech evidence, such as unreliable strength or unclear source.

[0061] By fusing semantic understanding, physical intensity, and spatial consistency to determine auditory risk scores, and by determining uncertain auditory risk scores, the problem of coarse-grained judgment patterns that rely solely on keyword matching or volume thresholds in related technologies is solved, effectively improving the accuracy of risk identification.

[0062] Step S504: Determine the second probability value corresponding to each of the plurality of preset risk type events based on the hearing risk score and the hearing risk uncertainty score.

[0063] In an exemplary embodiment, when the plurality of preset risk type events include risk events, risk-free events, and uncertain risk events, determining the second probability values ​​corresponding to the plurality of preset risk type events based on the auditory risk score and the auditory risk uncertainty score includes: determining a second value and a third difference between the second value and the auditory risk uncertainty score, and determining the second probability value corresponding to the risk event based on the product of the third difference and the auditory risk score, wherein the second value is used to indicate the value corresponding to the highest reliability of the auditory risk score; determining a second value and a fourth difference between the second value and the auditory risk score, and determining the second probability value corresponding to the risk-free event based on the product of the third difference and the fourth difference; and determining the second probability value corresponding to the uncertain risk event based on the auditory risk uncertainty score.

[0064] For the risk event (R) corresponding to the second probability value m a ({R}), calculate the second value and the hearing risk uncertainty score δ a The third difference 1-δ a The third difference represents the weight of evidence provided by the speech modality that, after excluding uncertainty, can be used to support risk assessment. The third difference is then compared with the auditory risk score r. a Multiplying them together yields the second probability value m of the risk event R. a ({R})=r a ·(1-δ a When the voice modality "has a significant risk intent" (r a "Higher" and "Higher consistency of evidence" (δ) a When m is lower a A higher ({R}) indicates a higher probability of risk. If r a High but δ a Also high (e.g., semantically clear but sound source mismatch), then (1-δ) a This will weaken the final probability and reduce the inaccuracy of risk assessment when evidence is insufficient.

[0065] For a risk-free event (¬R), the second probability value m a ({¬R}), calculate the second value and the hearing risk score r a The fourth difference 1-r a The fourth difference reflects the non-risk confidence that no obvious risk intent was detected in the speech modality. Multiplying the fourth difference by the third difference yields the second probability value m of the risk-free event ¬R. a ({¬R})=(1-r a )·(1-δ aWhen voice evidence does not demonstrate a risky intent (r a (Close to 0) and with no significant uncertainty (δ) a When r approaches 0, the "risk-free" state is determined to be valid. a Low but δ a High (e.g., unclear speech, unrecognizable), then m a ({¬R}) also approaches 0, indicating that "no risk" cannot be confirmed.

[0066] For an uncertain risk event (Θ), the second probability value m a (Θ), the hearing risk uncertainty score δ a directly used as the second probability value m of the uncertain event Θ a (Θ)=δ a This indicates the uncertainty of speech modality caused by semantic ambiguity, abnormal volume, spatial inconsistency, or noise interference.

[0067] In an exemplary embodiment, when the plurality of preset risk type events include risk events, risk-free events, and uncertain risk events, the risk score of the target area is determined based on the first probability value corresponding to each of the plurality of preset risk type events and the second probability value corresponding to each of the plurality of preset risk type events. This can be achieved through the following steps S602-S606:

[0068] Step S602: Determine the first conflict strength between the first probability value of the risk event and the second probability value of the risk-free event, and determine the second conflict strength between the second probability value of the risk event and the first probability value of the risk-free event.

[0069] The first conflict intensity is used to represent the conflict probability between a risky event recognized by vision and a risk-free event recognized by speech. The first conflict intensity is determined by the product of the first probability value of the risky event and the second probability value of the risk-free event, i.e., m. v ({R})·m a ({¬R}), the greater the first conflict strength, the more likely visual evidence supports the existence of risk, while audio evidence denies the risk. For example, a camera detects an elderly person falling (high m). v ({R})), but speech recognition did not detect a cry for help or the semantic meaning of "I'm fine" (high m) a ({¬R})).

[0070] The second conflict intensity is used to represent the conflict probability between a risky event recognized by speech recognition and a risk-free event recognized by vision. The second conflict intensity is determined by the product of the second probability value of the risky event and the first probability value of the risk-free event, i.e., m. a ({R})·m v({¬R}), the larger this value, the more likely the speech system has detected high-risk semantics (such as "help"), but the visual system has not clearly detected any abnormal targets (such as no human figure or falling posture). For example, a clear cry for help may be heard from inside the room (high m). a ({R})), but the camera failed to capture the target (height m) due to obstruction, backlighting, or angle issues. v ({¬R})).

[0071] Step S604: Determine the target conflict intensity based on the first conflict intensity and the second conflict intensity;

[0072] To comprehensively assess the overall inconsistency between the two modes, the first conflict intensity and the second conflict intensity are added together to obtain the target conflict intensity K. conflict :K conflict =m v ({R})·m a ({¬R})+m a ({R})·m v ({¬R}), K conflict ∈[0,1], indicating the overall conflict level between the visual and speech modalities in terms of risk and no risk. When K conflict When K = 0, it indicates that the visual modality and the speech modality are completely consistent (e.g., both support R or both support ¬R). conflict When the value approaches 1, it indicates a serious conflict between the visual and speech modalities.

[0073] Step S606: Determine the risk score of the target area based on the target conflict intensity, the first probability value corresponding to each of the multiple preset risk type events, and the second probability value corresponding to each of the multiple preset risk type events.

[0074] In an exemplary embodiment, determining a risk score for a target area based on the target conflict intensity, first probability values ​​corresponding to the plurality of preset risk type events, and second probability values ​​corresponding to the plurality of preset risk type events includes: determining a first synergy strength between the first probability value of the risk event and the second probability value of the risk event; determining a second synergy strength between the first probability value of the risk event and the second probability value of the uncertain risk event; determining a third synergy strength between the first probability value of the uncertain risk event and the second probability value of the determined risk event; determining the sum of the first synergy strength, the second synergy strength, and the third synergy strength; and determining a fifth difference between the third value and the target conflict intensity, wherein the third value indicates the maximum conflict intensity between the probability value of the risk event determined based on the image data and the probability value of the risk event determined based on the audio data; and determining the risk score based on the sum and the fifth difference.

[0075] The first coordination strength indicates the probability that a risk is confirmed based on both image and audio data. The first coordination strength is the product of the first probability value and the second probability value of the risk event, i.e., m. v ({R})·m a ({R}), for example, visual detection of falling posture (m v ({R})=0.85), speech recognition identified the semantic meaning of "help" (m a ({R})=0.92), indicating a high degree of synergy between the two.

[0076] The second coordination strength is used to indicate the probability that a risk is confirmed based on image data, but its existence is uncertain based on audio data. The second coordination strength is the product of the first probability value of the risk event and the second probability value of the uncertain risk event, i.e., m. v ({R})·m a (Θ), for example, the camera clearly captures an elderly person falling, but the speech fails to recognize the meaning due to environmental noise or slow speech speed. In this case, m a (Θ)>0 indicates that the speech modality cannot be determined, rather than being risk-free, effectively avoiding missed reports due to speech recognition failure.

[0077] The third cooperation strength is used to indicate the probability that the existence of a risk is uncertain based on image data but confirmed based on audio data. The third cooperation strength is the product of the first probability value of the uncertain risk event and the second probability value of the risk event, i.e., m. v (Θ)·m a ({R}), for example, a clear cry for help, "Fire!" can be heard from inside the room (m a ({R})=0.9), but the camera failed to detect flames or human bodies due to furniture obstruction (m v (Θ)=0.6). Thus, in the event of visual failure, an alarm is triggered based on high-confidence voice evidence, improving the accuracy of risk identification.

[0078] Add the first, second, and third coordination strengths together to obtain the sum, and then calculate the fifth difference (1-K) between the third value and the target conflict strength. conflict Finally, the risk score m is determined based on the sum and the fifth difference. comb ({R})=(m v ({R})·m a ({R})+m v ({R})·m a (Θ)+m v (Θ)·m a ({R}) / (1-K conflict ).

[0079] Step S206: Determine the target risk score of the target area based on the scoring correction factor and the risk score, wherein the scoring correction factor is used to indicate the impact of time and / or the scene type of the target area on the risk score.

[0080] In an exemplary embodiment, before determining the target risk score of the target area based on the rating correction factor and the risk score, the method further includes at least one of the following: determining the average risk score of the target area within a preset historical time period; determining the risk change rate of the target area based on the risk score and the average risk score; determining a historical risk trend factor based on the risk change rate; determining the duration of the target risk event when a target risk event is detected; determining a time correction factor based on the duration; and determining a scene correction factor based on the scene type of the target area, wherein the rating correction factor includes at least one of the following: the historical risk trend factor, the time correction factor, and the scene correction factor.

[0081] The risk score of the target area is continuously recorded at every moment within a preset historical time period (e.g., the past 30 seconds to 5 minutes), and its average risk score is calculated. Then, the currently calculated risk score m will be... comb The difference between ({R}) and the average risk score is used to obtain the risk change rate. The risk change rate reflects the dynamic trend of risk status over time. When ΔS > 0, it indicates that the risk is rising; when ΔS < 0, it indicates that the risk is easing. A hyperbolic tangent function is used to nonlinearly map the risk change rate, generating a historical risk trend factor T(t): T(t) = 1 + η·tanh(ΔS), where η > 0 is the trend control coefficient (e.g., η = 0.3), ensuring that T(t) ∈ [1-η, 1+η]. The historical risk trend factor automatically enhances the score when risk rises rapidly (T(t) > 1), and moderately suppresses fluctuations when risk falls (T(t) < 1), preventing overreaction to instantaneous noise or brief false triggers, where t represents the current moment.

[0082] Upon detecting the target risk event (i.e., m) comb When ({R}) exceeds a preset threshold, an event duration timer is activated to record the duration D of the target risk event from triggering to the present. Based on this duration, a time correction factor α(D) is determined: α(D) = 1 + λ·tanh(D / τ), where λ>0 is the cumulative gain coefficient and τ>0 is the time scale parameter. When D is short, α(D) is close to 1; when D is long (e.g., an elderly person falls and does not get up for 30 seconds), α(D) increases. This effectively distinguishes between false alarms and real, ongoing crises, enhancing the priority of response to real emergency events.

[0083] Furthermore, based on the scene type of the target area (e.g., bathroom, kitchen, bedroom, living room), the corresponding scene correction factor β(env) is retrieved from the preset mapping table: for example, in the bathroom scene, β... bath =1.2; In a kitchen setting, β kitchen =1.1; In the bedroom scenario, β bedroom =1.0; In the living room scenario, β living =0.9.

[0084] Finally, based on the above-mentioned scoring correction factors and risk scores, the target risk score S is determined. total :S total =β(env)·α(D)·T(t)·m comb ({R}), the target risk score ranges from [0, β_max·(1+λ)·(1+η)], and can be divided into multiple risk levels (such as L1 to L4) according to application requirements, which are used to trigger differentiated response strategies (such as notifying family members, automatic dialing, linking smoke detectors, activating emergency lighting, etc.).

[0085] Through the above steps, the visual and audio probability values ​​corresponding to multiple preset risk type events are determined based on image and audio data. The visual and audio probability values ​​are then fused to determine the risk score of the target area. The risk score is then corrected based on a score correction factor to obtain the target risk score. This solves the problem of inaccurate risk assessment caused by relying on single-modal perception technology in related technologies, thereby improving the accuracy of risk assessment in complex home environments.

[0086] In an exemplary embodiment, before determining the second probability values ​​corresponding to multiple preset risk type events based on the audio data of the target region, the method further includes: inputting the audio data into a speech classification model to identify abnormal audio data and normal audio data in the audio data; denoising the normal audio data using a noise reduction model and performing semantic parsing on the denoised normal audio data to determine the semantic information corresponding to the normal audio data; classifying the abnormal audio data to determine the abnormal type corresponding to the abnormal audio data; and determining the target semantic information of the audio data based on the semantic information and the abnormal type.

[0087] Audio data is input into a pre-trained speech classification model (such as Paddle Speech or a deep neural network-based audio classifier), which outputs two categories of results: anomalous audio data and normal audio data. Anomalous audio data refers to non-verbal sounds with clear danger characteristics, such as screams, breaking glass, impacts, fire explosions, and alarm sounds; normal audio data refers to non-emergency acoustic signals that can be identified as human speech, such as background conversations, television broadcasts, and radio audio. This classification step provides a semantic isolation basis for subsequent processing, effectively preventing normal speech from interfering with the identification of dangerous events.

[0088] For the identified normal audio data, it is input into an AI deep noise reduction model for speech enhancement. This AI deep noise reduction model, based on a frequency position coding and frequency domain-aware multi-head feature enhancer architecture, performs structured modeling of the time-frequency domain signal. While preserving the semantic integrity of the speech, it suppresses background noise, echo, and reverberation interference, reconstructing a clean speech spectrum with a high signal-to-noise ratio. The denoised audio data is then input into an ASR model (such as Whisper or DeepSpeech) to convert it into a text sequence. This text sequence is then fed into an LLM model for semantic parsing to extract its semantic intent. For example, it identifies high-risk semantics such as "help," "I fell," and "fire," or low-risk semantics such as "I'm okay" and "TV sound," ultimately outputting standardized semantic information.

[0089] For the identified abnormal audio data, the abnormal audio data is input into the abnormal audio classification sub-model. The abnormal audio classification sub-model performs fine-grained classification of abnormal sounds based on spectral features and temporal pattern recognition, and outputs the corresponding abnormal type, such as "scream", "explosion", "heavy object falling to the ground", "alarm sounding", etc.

[0090] Finally, the semantic information corresponding to normal audio data is fused with the anomaly type corresponding to abnormal audio data to determine the target semantic information. For example, if an anomaly type exists (such as "scream"), then that anomaly type is used as the target semantic information; if there is no anomaly type but semantic information exists (such as "help"), then that semantic information is used as the target semantic information; if both are missing or invalid, then the target semantic information is marked as having no clear semantic meaning.

[0091] In an exemplary embodiment, before denoising the normal audio data through a denoising model, the method further comprises: performing audio synthesis on clean sample audio data and noise sample audio data to obtain noisy audio data; performing audio framing on the noisy audio data with a target frame shift to determine a plurality of sub-noisy audio data; converting time-domain signals of the plurality of sub-noisy audio data into frequency-domain signals to determine a first spectral feature of the noisy audio data, wherein the first spectral feature is a complex domain representation, and the first spectral feature comprises a real part feature and an imaginary part feature; and performing model training on the denoising model based on the first spectral feature of the noisy audio data.

[0092] A set of signal-to-noise ratio sets is established according to actual application scenarios, wherein the signal-to-noise ratio set covers typical noise types in a home environment (such as TV sound, air conditioner sound, and wind sound) and different intensity levels (such as 5dB, 10dB, 15dB). According to the signal-to-noise ratio set, a speech sample is selected from a clean audio library, a corresponding noise sample is selected from a noise library, audio synthesis is performed on the clean speech sample and the noise sample to generate noisy audio data, and the time-domain representation of the noisy audio data is x∈R¹×M, where M is the total number of sampling points.

[0093] Audio framing processing is performed on the noisy audio data x described above, and the frame length W is set to 2 j , where j is a positive integer greater than 7, which ensures that the frequency domain resolution meets the requirements of speech spectrum analysis; the frame shift S is a preset target frame shift, which satisfies S<W and is usually taken as W / 2 or W / 2, so as to preserve signal continuity in the time domain and reduce inter-frame jumps. After framing, the noisy audio data is divided into N sub-noisy audio data, and each sub-noisy audio data is a short-time audio segment with a length of W.

[0094] Fast Fourier Transform (FFT for short) is performed on the time-domain signal of each sub-noisy audio data to map it from the time domain to the frequency domain. Since the FFT output has symmetric conjugation, only the first 1+W / 2 frequency points are retained to form a frequency-domain representation. Thus, the first spectral feature of the noisy audio data is obtained, which is a complex domain representation and denoted as X=X r +iX i , where the real part feature X r ∈R S×N , and the imaginary part feature X i ∈R S×N , the amplitude feature is |X|, the phase feature is α=arctan(X i / X r ), S=1+W / 2 is the number of frequency points, and N is the number of frames.

[0095] Simultaneously, the original clean audio y is subjected to the same frame segmentation and FFT processing as described above to obtain the corresponding clean spectral features Y=Y. r +iY i The real part is Y r ∈R S×N The imaginary part is Y i ∈R S×N The amplitude characteristic is |Y|, and the phase characteristic is α=arctan(Y). i / Y r The first spectral characteristic X=X with noise frequency. r +iX i As model input, with clean spectral features Y=Y r +iY i The denoising model is trained using this as the training target.

[0096] In an exemplary embodiment, training the denoising model based on the first spectral feature of the noisy frequency data includes: mapping the real part feature to a target dimension space based on the target operator of the denoising model, and extracting features from the real part feature mapped to the target dimension space through the convolutional layer of the denoising model to obtain a first real part feature; mapping the imaginary part feature to a target dimension space based on the target operator of the denoising model, and extracting features from the imaginary part feature mapped to the target dimension space through the convolutional layer of the denoising model to obtain a first imaginary part feature; adding frequency position encoding to the first real part feature to obtain a second real part feature, and adding frequency position encoding to the first imaginary part feature to obtain a second imaginary part feature; and training the denoising model based on the second real part feature and the second imaginary part feature.

[0097] The real part feature X from the first spectral feature of the noisy frequency r ∈R S×N (S is the number of frequency points, N is the number of frames) is mapped to a three-dimensional tensor X through the reshape operation. r ∈R 1×S×N This achieves structural reorganization from a two-dimensional spectrum to a channel dimension. Subsequently, a two-dimensional convolutional layer, Conv2D, with a 3×3 kernel and 1 padding, is applied. 1→C (Transforming the number of channels from 1 to C), map this tensor to a high-dimensional feature space and output the first real part feature. ∈R C×S×N Where C is the preset target dimension (e.g., C=64), and each time frequency point (S×N) is encoded as a C-dimensional feature vector.

[0098] Similarly, for the imaginary part feature X in the first spectral feature... i ∈R S×NIt employs the same structure and parameter sharing mechanism as the real part feature: firstly, it is mapped to X through the reshape operator. i ∈R 1×S×N Then through the same Conv2D 1→C The convolutional layer extracts features to obtain the corresponding first imaginary part features. ∈R C×S×N .

[0099] It should be noted that in related technologies, the noise reduction model directly accepts the feature sequence, while in the embodiments of this application, a convolutional layer is first used to map 1×S×N to C×S×N, and then global attention is performed after introducing a local receptive field.

[0100] After obtaining the first real part feature Features of the first imaginary part Frequency location coding is then introduced, and the frequency location coding is used to define the learnable parameters P∈R of the model. C×S×1 It varies only along the frequency dimension S, is shared across all time frames N, and is used to explicitly model the physical order relationship between frequency bands (such as the continuity from low to high frequencies). This encoded P is broadcast to all N time frames, generating P broadcast ∈R C×S×N Subsequently, it is added to the first real part feature and the first imaginary part feature respectively to obtain the second real part feature. = +P broadcast Second imaginary part features = +P broadcast ,in, ∈R C×S×N , ∈R C×S×N The denoising model is trained based on the second real part feature and the second imaginary part feature.

[0101] It should be noted that the standard Transformer encodes the syntactic or semantic order of tokens in a speech sequence, using one-dimensional positional encoding, which is applied independently only in the sequence dimension to represent the linear arrangement of elements in the sequence. In contrast, the encoding object in this embodiment is the physical frequency order of frequency points in the speech spectrum; the encoding dimension is three-dimensional, with independent processing of the channel axis, frequency axis, and time axis. The frequency axis uses independent positional encoding, while the time axis uses a shared encoding mechanism.

[0102] In the standard Transformer, the attention scope is limited to sequence elements within the same modality. Queries, keys, and values ​​all originate from the same input sequence, and the attention dimension is the pairwise relationship between elements within a one-dimensional sequence. In contrast, this embodiment uses a frequency axis for the query vector of the attention mechanism, while keys and values ​​originate from the entire time-frequency joint space (i.e., a combination of all time frames and all frequency points). Regarding the multi-head attention design, different attention heads in this embodiment can learn and focus on specific frequency bands (such as low-frequency, mid-frequency, and high-frequency bands) to adapt to the acoustic characteristics and noise interference patterns of different frequency bands in the speech signal; whereas the attention heads in the standard Transformer have no physical correspondence. This embodiment not only significantly improves the speech feature separation capability in low signal-to-noise ratio environments but also enhances the interpretability of the model, i.e., understanding the model's behavior; simultaneously, it improves noise reduction performance: different frequency bands have different noise characteristics, and multi-head division of labor allows for targeted processing.

[0103] In an exemplary embodiment, training the denoising model based on the second real part feature and the second imaginary part feature includes: flattening the second real part feature and the second imaginary part feature into a third real part feature and a third imaginary part feature; performing feature enhancement on the third real part feature and the third imaginary part feature through the multi-head attention module of the denoising model to determine a first enhanced feature corresponding to the third real part feature and a second enhanced feature corresponding to the third imaginary part feature; and training the denoising model based on the first enhanced feature and the second enhanced feature.

[0104] After obtaining the second real part feature ∈R C×S×N Features of the second imaginary part ∈R C×S×N Then, a flattening operation is performed on both in the spatial dimension (i.e., frequency point dimension S and time frame dimension N). Specifically, the flattening operation is performed on both. Flattening the spatial dimension (S×N) into a one-dimensional sequence yields the third real part feature. ∈R C×L Where L = S × N is the total number of positions after flattening; similarly, for Performing the same operation yields the third imaginary part feature. ∈R C×L This flattening operation does not change the number of feature channels C, but only transforms the two-dimensional time-frequency structure S×N into a one-dimensional sequence L, providing a standard input format for subsequent multi-head attention mechanisms.

[0105] The third real part feature and the third imaginary part feature Independent and structurally symmetrical multi-head attention modules are constructed to achieve semantic enhancement across time frequencies. Specifically, for the third real part feature... ∈R C×L Set the number of attention heads H, and the dimension d of each head. h =C / H, where a learnable projection matrix is ​​defined for each head h (1≤h≤H). , , Calculate query Q h = ∈R dh×L , key K h = ∈R dh×L Value V h = ∈R dh×L Calculate attention weight A h =Softmax( )∈R L×L Characterizes the correlation between any two time-frequency points, and outputs header O. h =V h ∈R dh×L To achieve context-aware enhancement for each time-frequency point, all H header outputs are concatenated: O = Concat(O1,...,O) H )∈R C ×L Through the learnable weight matrix W O Perform a linear transformation: =W O O, then based on the reshape operation, C×L is rearranged into C×S×N to obtain the first enhanced feature. ∈R C×S×N Similarly, for the third imaginary part feature... ∈R C×L Perform the exact same multi-head attention process to obtain the second enhanced feature. ∈R C×S×N The denoising model is trained based on the first and second enhancement features.

[0106] In an exemplary embodiment, training the denoising model based on the first enhancement feature and the second enhancement feature includes: calculating the first enhancement feature and the second enhancement feature using the residual and normalization module of the denoising model to obtain a second spectral feature of the denoised audio data; determining a third spectral feature of the clean sample audio data; determining a loss function of the denoising model based on the second spectral feature and the third spectral feature; and adjusting the learning parameters of the denoising model based on the loss function, wherein the loss function includes at least one of the following: a correlation spectrum distortion loss function, a signal-to-noise ratio loss function, a perceptual speech quality assessment loss function, and a speech activity detection loss function.

[0107] After obtaining the first enhanced feature ∈R C×S×N With the second enhancement feature ∈R C×S×N Then, deep processing is performed on both through residual and normalization modules. Specifically, the first enhanced feature... Perform residual join, and transfer the second real part feature Adding it to the intermediate features yields the intermediate features. Subsequently, the intermediate feature is normalized. =LayerNorm ( Similarly, for the second enhancement feature... Perform the same operation to obtain =LayerNorm ( ).according to and Determine the second spectral characteristics.

[0108] Normalized and The input is fed into a residual network, which consists of B (e.g., 6) stacked residual blocks. Each residual block is designed as follows: Z = ReLU(Conv2D) C→C (X)), X out =X+Conv2D C→C (Z), after processing by the B-layer residual blocks, outputs higher-order semantic features. ∈R C×S×N , ∈R C×S×N Finally, two independent 1×1 convolutional layers are used to restore the real and imaginary features to the original spectral dimension, generating the real and imaginary parts of the third spectral feature of the clean sample audio data, i.e., A. R =Conv2D C→1 ( )∈R S×N A i =Conv2D C→1 ( )∈R S×N .

[0109] The loss function of the denoising model is determined based on the second and third spectral features, including: determining the correlation spectral distortion loss function L according to the following formula. RSD = And determine the signal-to-noise ratio loss function L according to the following formula. SNR =-10log 10 ( The perceptual speech quality assessment loss function L is determined according to the following formula. PESQ=-PESQ(A,Y), and determine the speech activity detection loss function according to the following formula. Finally, the loss function L was determined. total =λ1·L RSD +λ2·L SNR +λ3·L PESQ +λ4·L vad Among them, Voice Activity Detection (VAD) is used to predict whether there is speech in each frame, and the true voice activity label of the nth frame is... ∈{0,1}, where 1 indicates the presence of speech and 0 indicates the absence of speech; Let be the predicted speech probability value of the model output for the nth frame, when If _vad(n) ≥ 0.5, it is determined to be a speech frame; otherwise, it is a non-speech frame. The frame is then determined using the backpropagation algorithm based on L. total The model training is completed by iteratively optimizing all learnable parameters of the denoising model (including convolution kernel, attention projection matrix, frequency position encoding parameters, normalization scaling factor, etc.) until the loss converges.

[0110] The noise reduction model in the embodiments of this application can solve the noise interference problem.

[0111] To better understand the process of determining the risk score described above, the following description, in conjunction with optional embodiments, further illustrates the implementation flow of the risk score determination method, but is not intended to limit the technical solutions of the embodiments of this application.

[0112] Figure 3 This is a schematic diagram of the architecture of a risk scoring method according to an optional embodiment of this application, such as... Figure 3 As shown, it includes the following:

[0113] Image data of the target area is acquired in real time using visual sensors such as cameras, and audio data of the target area is acquired in real time using audio sensors such as microphones. The image data is input into a CV model library, which performs visual modality analysis on the image data to obtain the visual modality analysis results. The specific analysis process is as follows: Figure 4 As shown.

[0114] Simultaneously, the audio data is input into a speech model library. Using speech classification and recognition models from this library, as well as speech recognition models from a Natural Language Processing (NLP) model library, the audio data undergoes AI-based deep noise reduction technology-based speech modality analysis to obtain the target semantic information of the audio data. The specific analysis process is as follows: Figure 5 As shown.

[0115] Risk scoring is performed on visual modality parsing results and target semantic information using a multimodal spatiotemporal risk cognition kernel algorithm, specifically including the following steps:

[0116] Step 1: Visual modality risk calculation;

[0117] The process involves acquiring a first confidence level for a risky target (e.g., a knife, flame, snake, etc.) in the image data and a second confidence level for an abnormal pose of the target object. A weighted fusion of the first and second confidence levels yields a visual risk score for the visual modality. To quantify the uncertainty of visual perception, a visual risk uncertainty score is determined based on the average of the first and second confidence levels. Finally, first probability values ​​for multiple preset risk type events are determined based on the visual risk score and the visual uncertainty score.

[0118] Step 2: Voice modality risk calculation;

[0119] A sound source consistency score for the target area is determined based on the auditory azimuth of the sound source and the visual azimuth of the target object. This score measures whether the sound source spatially matches the visual target. Further, a risk level score for the target object is determined based on the target semantic information from the audio data, which is obtained from the audio data parsing results. Simultaneously, the decibel information of the audio data is normalized to obtain a decibel score for the target area. Based on the risk level score, decibel score, and sound source consistency score, an auditory risk score and an auditory risk uncertainty score for the target area are determined. Then, based on these scores, second probability values ​​corresponding to multiple preset risk type events are determined.

[0120] Step 3: Multimodal evidence fusion;

[0121] A conflict factor is determined based on the product of a first probability value of a risky event and a second probability value of a risk-free event, and the product of the second probability value of a risky event and the first probability value of a risk-free event. Multiple preset risk type events include: risky events, risk-free events, and uncertain risk events. A risk score for the target area is determined based on the conflict factor, multiple first probability values, and multiple second probability values.

[0122] Step 4: Set historical risk trend factors;

[0123] The risk change rate of the target area is determined based on the average risk score of the target area over the past period and the risk score in step three, and the historical risk trend factor is determined based on the risk change rate.

[0124] Step 5: Set the time accumulation factor and scene correction factor;

[0125] The time correction factor (equivalent to the time accumulation factor) is determined based on the duration of the risk event, and the scenario correction factor is determined based on the scenario type of the target area.

[0126] Step Six: Determine the final risk score;

[0127] The final risk score for the target area is determined based on historical risk trend factors, time correction factors, scenario correction factors, and risk scores.

[0128] After the multimodal spatiotemporal risk cognition kernel completes the risk scoring for both visual and speech modalities, the trusted evidence chain generator for audio-visual spatiotemporal alignment calculates the azimuth difference between the auditory azimuth of the sound source in the target area and the visual azimuth of the target object, and simultaneously determines the audio-visual time difference between the audio and image data. If the azimuth difference is less than the azimuth threshold and the audio-visual time difference is less than the time difference threshold, the audio and image data are stored in both the cloud device and the local terminal device; otherwise, they are stored only in the cloud device.

[0129] Figure 4 This is a schematic diagram of the visual modality analysis module of a risk scoring determination method according to an embodiment of the present invention, as shown below. Figure 4 As shown, it includes the following:

[0130] The visual modality analysis results are obtained by sequentially performing scene classification, object detection, and human pose determination tasks using detection / classification models and pose determination models from the CV model library. The classification model can be an image classification model based on a convolutional neural network (CNN) or Transformer architecture to identify the current environment type (e.g., bedroom, kitchen, living room). The detection model can use mainstream object detection algorithms such as YOLO series, Faster R-CNN, and SSD, outputting the detected objects and their bounding box coordinates and confidence scores in the image. Identified objects include, but are not limited to, dangerous targets such as knives, flames, and snakes. The pose determination model can be a human keypoint detection model (e.g., PP-TinyPose, OpenPose) to extract human joint position information and determine the presence of abnormal postures (e.g., tilting, lying down, violent struggling).

[0131] The three types of models mentioned above can be dynamically selected and combined according to the computing power limitations, power consumption requirements and real-time requirements of the actual deployment environment. Their output results are uniformly and structurally stored in the visual modality information database table (Table 1). The fields include timestamp, scene type, list of detected targets and their confidence, human ID and corresponding pose information, etc., providing high-precision and structured visual semantic input for subsequent multimodal fusion and risk cognition.

[0132] Table 1 Visual Modal Information Database Table

[0133]

[0134] Figure 5 This is a schematic diagram of the speech modality parsing module of a risk scoring determination method according to an embodiment of the present invention, as shown below. Figure 5 As shown, it includes the following:

[0135] When audio is input into the speech model library, the speech classification model in the speech model library (such as PaddleSpeech, AudioSet, etc.) performs a preliminary classification of the audio (equivalent to speech binary classification in the figure), distinguishing abnormal audio (such as sudden dangerous sounds such as screams, explosions, glass breaking, and calls for help) from normal speech (such as non-emergency sounds such as conversations, TV playback, and background music).

[0136] For audio segments identified as abnormal, multi-classification of speech is performed to determine the specific event type (such as screaming, distress calls, impacts, fire alarms, etc.), and category labels and confidence scores are output. For audio segments identified as normal speech, speech enhancement processing is performed using AI deep noise reduction technology. Specifically, based on a frequency domain context-aware architecture, the speech spectrum is reconstructed in a structured manner through frequency position coding and a frequency-aware multi-head feature enhancer, effectively suppressing background noise interference and improving speech clarity and recognizability.

[0137] The denoised, clean speech signal is then fed into an ASR model (such as the ASR engine of Whisper, DeepSpeech, or PaddleSpeech) to convert it into text. The generated text content is further fed into an NLP model (e.g., a large language model LLM) for semantic parsing to determine the semantic information corresponding to the normal audio data.

[0138] Finally, the target semantic information of the audio data is determined based on the semantic information and the anomaly type of the abnormal audio. The target semantic information includes the timestamp, volume, semantic information, and anomaly category label. These are uniformly and structurally stored in the speech modality database table (Table 2) to form a high-precision, semantic auditory cognitive output, providing a reliable basis for subsequent cross-modal risk fusion.

[0139] Table 2 Speech Modality Database Tables

[0140]

[0141] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0142] This embodiment also provides a risk scoring determination device for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0143] Figure 6 This is a structural block diagram of a risk scoring determination device according to an embodiment of this application, such as... Figure 6 As shown, the device includes:

[0144] The first determining module 62 is used to determine a first probability value corresponding to a plurality of preset risk type events based on image data of the target area, and to determine a second probability value corresponding to a plurality of preset risk type events based on audio data of the target area.

[0145] The second determining module 64 is used to determine the risk score of the target area based on the first probability value corresponding to the plurality of preset risk type events and the second probability value corresponding to the plurality of preset risk type events.

[0146] The third determining module 66 is used to determine the target risk score of the target area based on the scoring correction factor and the risk score, wherein the scoring correction factor is used to indicate the impact of time and / or the scene type of the target area on the risk score.

[0147] The aforementioned device determines the visual and audio probability values ​​corresponding to multiple preset risk type events based on image and audio data. It then fuses the visual and audio probability values ​​to determine the risk score of the target area and corrects the risk score based on a score correction factor to obtain the target risk score. This solves the problem of inaccurate risk assessment caused by relying on single-modal perception technology in related technologies, thereby improving the accuracy of risk assessment in complex home environments.

[0148] In an exemplary embodiment, the first determining module 62 is further configured to acquire a first confidence level of a risk target in the image data and a second confidence level of an abnormal posture of a target object in the image data, wherein the risk target is a biological individual or a non-biological individual, and the target object is a biological individual; determine a visual risk score and a visual risk uncertainty score for the target area based on the first confidence level and the second confidence level, wherein the visual risk score is used to indicate the probability of a risk event occurring in the target area determined based on the image data, and the visual risk uncertainty score is used to indicate the degree of unreliability of the visual risk score; and determine first probability values ​​corresponding to the plurality of preset risk type events based on the visual risk score and the visual risk uncertainty score.

[0149] In an exemplary embodiment, the first determining module 62 is further configured to determine the visual risk score based on the product of a first weight and the first confidence level, the product of a second weight and the second confidence level, and the product of a third weight and the first confidence level and the second confidence level, wherein the first weight is the weight corresponding to the first confidence level, the second weight is the weight corresponding to the second confidence level, and the third weight is the weight corresponding to the product of the first confidence level and the second confidence level; and to determine the average value of the sum of the first confidence level and the second confidence level, and to determine the visual risk uncertainty score based on the difference between a first value and the average value, wherein the first value is used to indicate the value corresponding to the highest reliability level of the visual risk score.

[0150] In an exemplary embodiment, the first determining module 62 is further configured to: determine a first difference between a first numerical value and the visual risk uncertainty score; determine a first probability value corresponding to the risk event based on the product of the first difference and the visual risk score, wherein the first numerical value is used to indicate the numerical value corresponding to the highest reliability of the visual risk score; determine a second difference between the first numerical value and the visual risk score; determine a first probability value corresponding to the risk-free event based on the product of the first difference and the second difference; and determine a first probability value corresponding to the uncertain risk event based on the visual risk uncertainty score.

[0151] In an exemplary embodiment, the first determining module 62 is further configured to: determine a sound source consistency score of the target region based on the audio data and the image data; determine target semantic information of the audio data based on the parsing result of the audio data; determine a risk level score of the target object in the target region based on the target semantic information; and normalize the decibel information of the audio data to determine a decibel score of the target region; and determine second probability values ​​corresponding to the plurality of preset risk type events based on the risk level score, the decibel score, and the sound source consistency score.

[0152] In an exemplary embodiment, the first determining module 62 is further configured to perform target detection on the audio data to determine the auditory azimuth angle of the target object in the target area, and to perform target detection on the image data to determine the visual azimuth angle of the target object in the target area; and to determine the sound source consistency score based on the auditory azimuth angle and the visual azimuth angle.

[0153] In an exemplary embodiment, the first determining module 62 is further configured to determine the azimuth difference between the auditory azimuth and the visual azimuth, and to determine the audio-visual time difference between the audio data and the image data; and to store the audio data and the image data if the azimuth difference is less than an azimuth threshold and the audio-visual time difference is less than a time difference threshold.

[0154] In an exemplary embodiment, the first determining module 62 is further configured to determine an auditory risk score and an auditory risk uncertainty score for the target area based on the risk level score, the decibel score, and the sound source consistency score; and to determine second probability values ​​corresponding to the plurality of preset risk type events based on the auditory risk score and the auditory risk uncertainty score.

[0155] In an exemplary embodiment, the first determining module 62 is further configured to determine the hearing risk score based on the product of a fourth weight and the risk level score, the product of a fifth weight and the decibel score, the product of a sixth weight and the risk level score and the decibel score, and the product of a seventh weight and the sound source consistency score; and to determine the hearing risk uncertainty score based on the average of the risk level score, the decibel score and the sound source consistency score.

[0156] In one exemplary embodiment, the first determining module 62 is further configured to: determine a second value and a third difference between the second value and the hearing risk uncertainty score; determine a second probability value corresponding to the risk event based on the product of the third difference and the hearing risk score, wherein the second value is used to indicate the value corresponding to the highest reliability of the hearing risk score; determine a fourth difference between the second value and the hearing risk score; determine a second probability value corresponding to the risk-free event based on the product of the third difference and the fourth difference; and determine a second probability value corresponding to the uncertain risk event based on the hearing risk uncertainty score.

[0157] In an exemplary embodiment, the second determining module 64 is further configured to determine a first conflict intensity between the first probability value of the risk event and the second probability value of the risk-free event, and to determine a second conflict intensity between the second probability value of the risk event and the first probability value of the risk-free event; determine a target conflict intensity based on the first conflict intensity and the second conflict intensity; and determine a risk score for the target area based on the target conflict intensity, the first probability values ​​corresponding to the plurality of preset risk type events, and the second probability values ​​corresponding to the plurality of preset risk type events.

[0158] In one exemplary embodiment, the second determining module 64 is further configured to determine a first coordination strength between a first probability value of the risk event and a second probability value of the risk event, a second coordination strength between the first probability value of the risk event and a second probability value of the uncertain risk event, and a third coordination strength between the first probability value of the uncertain risk event and the second probability value of the determined risk event; determine the sum of the first coordination strength, the second coordination strength, and the third coordination strength, and determine a fifth difference between the third value and the target conflict strength, wherein the third value is used to indicate the maximum conflict strength between the probability value of the risk event determined based on the image data and the probability value of the risk event determined based on the audio data; and determine the risk score based on the sum and the fifth difference.

[0159] In an exemplary embodiment, the third determining module 66 is further configured to: determine the average risk score of the target area within a preset historical time period; determine the risk change rate of the target area based on the risk score and the average risk score; determine a historical risk trend factor based on the risk change rate; determine the duration of the target risk event when a target risk event is detected; determine a time correction factor based on the duration; and determine a scene correction factor based on the scene type of the target area, wherein the score correction factor includes at least one of the following: the historical risk trend factor, the time correction factor, and the scene correction factor.

[0160] In an exemplary embodiment, the first determining module 62 is further configured to input the audio data into a speech classification model to determine abnormal audio data and normal audio data in the audio data; perform noise reduction on the normal audio data using a noise reduction model, and perform semantic parsing on the noise-reduced normal audio data to determine the semantic information corresponding to the normal audio data; classify the abnormal audio data to determine the abnormal type corresponding to the abnormal audio data; and determine the target semantic information of the audio data based on the semantic information and the abnormal type.

[0161] In an exemplary embodiment, the first determining module 62 is further configured to perform audio synthesis on clean sample audio data and noisy sample audio data to obtain noisy frequency data; perform audio framing on the noisy frequency data with a target frame shift to determine multiple sub-band noisy frequency data; convert the time-domain signal of the multiple sub-band noisy frequency data into a frequency-domain signal to determine the first spectral feature of the noisy frequency data, wherein the first spectral feature is a complex domain expression and includes: real part feature and imaginary part feature; and perform model training on the noise reduction model based on the first spectral feature of the noisy frequency data.

[0162] In an exemplary embodiment, the first determining module 62 is further configured to: map the real part features to a target dimension space based on the target operator of the denoising model; and extract features from the real part features mapped to the target dimension space through the convolutional layer of the denoising model to obtain a first real part feature; map the imaginary part features to a target dimension space based on the target operator of the denoising model; and extract features from the imaginary part features mapped to the target dimension space through the convolutional layer of the denoising model to obtain a first imaginary part feature; add frequency position encoding to the first real part feature to obtain a second real part feature; and add frequency position encoding to the first imaginary part feature to obtain a second imaginary part feature; and train the denoising model based on the second real part feature and the second imaginary part feature.

[0163] In an exemplary embodiment, the first determining module 62 is further configured to flatten the second real part feature and the second imaginary part feature into a third real part feature and a third imaginary part feature; perform feature enhancement on the third real part feature and the third imaginary part feature through the multi-head attention module of the denoising model to determine a first enhanced feature corresponding to the third real part feature and a second enhanced feature corresponding to the third imaginary part feature; and perform model training on the denoising model based on the first enhanced feature and the second enhanced feature.

[0164] In an exemplary embodiment, the first determining module 62 is further configured to calculate the first enhancement feature and the second enhancement feature using the residual and normalization module of the noise reduction model to obtain the second spectral feature of the noise-reduced audio data; determine the third spectral feature of the clean sample audio data; determine the loss function of the noise reduction model based on the second spectral feature and the third spectral feature; and adjust the learning parameters of the noise reduction model based on the loss function, wherein the loss function includes at least one of the following: correlation spectrum distortion loss function, signal-to-noise ratio loss function, perceptual speech quality assessment loss function, and speech activity detection loss function.

[0165] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination.

[0166] Embodiments of the present invention also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to perform the steps in any of the above method embodiments when executed.

[0167] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0168] Embodiments of the present invention also provide an electronic device including a memory and a processor, the memory storing a computer program and the processor being configured to run the computer program to perform the steps in any of the above method embodiments.

[0169] In one exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0170] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0171] Embodiments of this application also provide a computer program product, including a computer program that, when executed by a processor, implements the steps of the methods described in various embodiments of this application.

[0172] Embodiments of this application also provide a computer program product, including a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the methods described in various embodiments of this application.

[0173] It is obvious to those skilled in the art that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those described herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0174] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for determining a risk score, characterized in that, include: Based on the image data of the target area, a first probability value corresponding to a plurality of preset risk type events is determined, and based on the audio data of the target area, a second probability value corresponding to a plurality of preset risk type events is determined. The risk score of the target area is determined based on the first probability value corresponding to the plurality of preset risk type events and the second probability value corresponding to the plurality of preset risk type events. A target risk score for the target area is determined based on a scoring correction factor and the risk score, wherein the scoring correction factor is used to indicate the impact of time and / or the scene type of the target area on the risk score; Where the plurality of preset risk type events include risk events, risk-free events, and uncertain risk events, determining the risk score of the target area based on the first probability value and the second probability value corresponding to the plurality of preset risk type events respectively includes: determining a first conflict intensity between the first probability value of the risk event and the second probability value of the risk-free event; and determining a second conflict intensity between the second probability value of the risk event and the first probability value of the risk-free event; determining a target conflict intensity based on the first conflict intensity and the second conflict intensity; and determining the risk score of the target area based on the target conflict intensity, the first probability value and the second probability value corresponding to the plurality of preset risk type events respectively. The method of determining first probability values ​​corresponding to multiple preset risk type events based on image data of the target area includes: acquiring a first confidence level of a risk target in the image data and a second confidence level of an abnormal posture of a target object in the image data, wherein the risk target is a biological individual or a non-biological individual, and the target object is a biological individual; determining a visual risk score and a visual risk uncertainty score for the target area based on the first confidence level and the second confidence level, wherein the visual risk score indicates the probability of a risk event occurring in the target area determined based on the image data, and the visual risk uncertainty score indicates the degree of unreliability of the visual risk score; and determining the first probability values ​​corresponding to the multiple preset risk type events based on the visual risk score and the visual risk uncertainty score. The method of determining second probability values ​​corresponding to multiple preset risk type events based on audio data of the target area includes: determining a sound source consistency score of the target area based on the audio data and the image data; determining target semantic information of the audio data based on the parsing results of the audio data; determining a risk level score of the target object in the target area based on the target semantic information; and normalizing the decibel information of the audio data to determine a decibel score of the target area; and determining the second probability values ​​corresponding to the multiple preset risk type events based on the risk level score, the decibel score, and the sound source consistency score.

2. The method according to claim 1, characterized in that, The visual risk score and visual risk uncertainty score of the target area are determined based on the first confidence level and the second confidence level, including: The visual risk score is determined based on the product of a first weight and the first confidence level, the product of a second weight and the second confidence level, and the product of a third weight and the first and second confidence levels, wherein the first weight is the weight corresponding to the first confidence level, the second weight is the weight corresponding to the second confidence level, and the third weight is the weight corresponding to the product of the first and second confidence levels; and, The average of the first confidence level and the second confidence level is determined, and the visual risk uncertainty score is determined based on the difference between the first value and the average value, wherein the first value is used to indicate the value corresponding to the highest reliability of the visual risk score.

3. The method according to claim 1, characterized in that, When the plurality of preset risk type events include risk events, risk-free events, and uncertain risk events, the first probability value corresponding to each of the plurality of preset risk type events is determined based on the visual risk score and the visual risk uncertainty score, including: A first difference between a first numerical value and the visual risk uncertainty score is determined, and a first probability value corresponding to the risk event is determined based on the product of the first difference and the visual risk score, wherein the first numerical value is used to indicate the value corresponding to the highest reliability of the visual risk score; Determine the difference between the first numerical value and the second visual risk score, and determine the first probability value corresponding to the risk-free event based on the product of the first difference and the second difference; and, The first probability value corresponding to the uncertain risk event is determined based on the visual risk uncertainty score.

4. The method according to claim 1, characterized in that, Determining the sound source consistency score of the target region based on the audio data and the image data includes: Target detection is performed on the audio data to determine the auditory azimuth angle of the target object in the target region, and, Target detection is performed on the image data to determine the visual azimuth angle of the target object in the target region; The sound source consistency score is determined based on the auditory azimuth and the visual azimuth.

5. The method according to claim 4, characterized in that, After performing target detection on the audio data to determine the auditory azimuth angle of the target object in the target region, and performing target detection on the image data to determine the visual azimuth angle of the target object in the target region, the method further includes: Determine the azimuth difference between the auditory azimuth and the visual azimuth, and determine the audio-visual time difference between the audio data and the image data; If the azimuth angle difference is less than the azimuth angle threshold and the audio-visual time difference is less than the time difference threshold, the audio data and the image data are stored.

6. The method according to claim 1, characterized in that, The second probability value corresponding to each of the plurality of preset risk type events is determined based on the risk level score, the decibel score, and the sound source consistency score, including: The hearing risk score and hearing risk uncertainty score of the target area are determined based on the risk level score, the decibel score, and the sound source consistency score. The second probability value corresponding to each of the plurality of preset risk type events is determined based on the hearing risk score and the hearing risk uncertainty score.

7. The method according to claim 6, characterized in that, The hearing risk score and hearing risk uncertainty score for the target area are determined based on the risk level score, the decibel score, and the sound source consistency score, including: The hearing risk score is determined based on the product of the fourth weight and the risk level score, the product of the fifth weight and the decibel score, the product of the sixth weight and the risk level score and the decibel score, and the seventh weight and the sound source consistency score; and, The hearing risk uncertainty score is determined based on the average of the risk level score, the decibel score, and the sound source consistency score.

8. The method according to claim 6, characterized in that, When the plurality of preset risk type events include risk events, risk-free events, and uncertain risk events, the second probability value corresponding to each of the plurality of preset risk type events is determined based on the hearing risk score and the hearing risk uncertainty score, including: Determine the second value and the third difference between the hearing risk uncertainty score, and determine the second probability value corresponding to the risk event based on the product of the third difference and the hearing risk score, wherein the second value is used to indicate the value corresponding to the highest reliability of the hearing risk score; Determine the second value and the fourth difference between the hearing risk score, and determine the second probability value corresponding to the risk-free event based on the product of the third difference and the fourth difference; and, The second probability value corresponding to the uncertain risk event is determined based on the hearing risk uncertainty score.

9. The method according to claim 1, characterized in that, The risk score of the target area is determined based on the target conflict intensity, the first probability values ​​corresponding to the plurality of preset risk type events, and the second probability values ​​corresponding to the plurality of preset risk type events, including: Determine a first cooperative strength between a first probability value of the risk event and a second probability value of the risk event, determine a second cooperative strength between the first probability value of the risk event and the second probability value of the uncertain risk event, and determine a third cooperative strength between the first probability value of the uncertain risk event and the second probability value of the determined risk event; The sum of the first coordination strength, the second coordination strength, and the third coordination strength is determined, and a fifth difference between the third value and the target conflict strength is determined, wherein the third value is used to indicate the maximum conflict strength between the probability value of the risk event determined based on the image data and the probability value of the risk event determined based on the audio data. The risk score is determined based on the sum and the fifth difference.

10. The method according to claim 1, characterized in that, Before determining the target risk score for the target area based on the scoring correction factor and the risk score, the method further includes at least one of the following: Determine the average risk score of the target area within a preset historical time period, determine the risk change rate of the target area based on the risk score and the average risk score, and determine the historical risk trend factor based on the risk change rate; If a target risk event is detected, the duration of the target risk event is determined, and a time correction factor is determined based on the duration. A scenario correction factor is determined based on the scenario type of the target area, wherein the scoring correction factor includes at least one of the following: the historical risk trend factor, the time correction factor, and the scenario correction factor.

11. The method according to claim 1, characterized in that, Before determining the second probability values ​​corresponding to multiple preset risk type events based on the audio data of the target area, the method further includes: The audio data is input into a speech classification model to identify abnormal and normal audio data. The normal audio data is denoised using a denoising model, and the denoised normal audio data is semantically parsed to determine the semantic information corresponding to the normal audio data. The abnormal audio data is also classified to determine the abnormal type corresponding to the abnormal audio data. The target semantic information of the audio data is determined based on the semantic information and the anomaly type.

12. The method according to claim 11, characterized in that, Before denoising the normal audio data using a denoising model, the method further includes: Audio synthesis is performed on clean sample audio data and noisy sample audio data to obtain noisy audio data; The noisy frequency data is divided into audio frames based on the target frame shift to determine multiple sub-band noisy frequency data. The time-domain signal of the plurality of sub-band noise frequency data is converted into a frequency-domain signal to determine the first spectral feature of the noise frequency data, wherein the first spectral feature is expressed in the complex domain and includes: real part feature and imaginary part feature; The noise reduction model is trained based on the first spectral features of the noisy frequency data.

13. The method according to claim 12, characterized in that, The noise reduction model is trained based on the first spectral features of the noisy frequency data, including: The real part features are mapped to the target dimension space by the target operator of the denoising model, and the real part features mapped to the target dimension space are extracted by the convolutional layer of the denoising model to obtain the first real part features; and the imaginary part features are mapped to the target dimension space by the target operator of the denoising model, and the imaginary part features mapped to the target dimension space are extracted by the convolutional layer of the denoising model to obtain the first imaginary part features. Frequency position encoding is added to the first real part feature to obtain the second real part feature, and frequency position encoding is added to the first imaginary part feature to obtain the second imaginary part feature; The noise reduction model is trained based on the second real part feature and the second imaginary part feature.

14. The method according to claim 13, characterized in that, The denoising model is trained based on the second real part feature and the second imaginary part feature, including: Flatten the second real part feature and the second imaginary part feature into a third real part feature and a third imaginary part feature; The multi-head attention module of the noise reduction model is used to enhance the third real part feature and the third imaginary part feature to determine the first enhanced feature corresponding to the third real part feature and the second enhanced feature corresponding to the third imaginary part feature; The noise reduction model is trained based on the first enhancement feature and the second enhancement feature.

15. The method according to claim 14, characterized in that, The noise reduction model is trained based on the first enhancement feature and the second enhancement feature, including: The first enhancement feature and the second enhancement feature are calculated using the residual and normalization module of the noise reduction model to obtain the second spectral feature of the noise-reduced audio data. Determine the third spectral feature of the clean sample audio data; The loss function of the noise reduction model is determined based on the second spectral feature and the third spectral feature, and the learning parameters of the noise reduction model are adjusted based on the loss function. The loss function includes at least one of the following: correlation spectrum distortion loss function, signal-to-noise ratio loss function, perceptual speech quality assessment loss function, and speech activity detection loss function.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the method described in any one of claims 1 to 15.

17. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method described in any one of claims 1 to 15.

18. A computer program product comprising a computer program that, when executed by a processor, implements the steps of the method described in any one of claims 1 to 15.

Images