Potential danger identification system based on environmental perception

Through the potential hazard identification system based on environment perception, the continuous monitoring and analysis of image and audio data can be used to identify and screen potential hazard subjects, solving the problem of insufficient real-time due to data processing delays in complex environments by traditional equipment, and improving the problem of insufficient real-time performance caused by data processing delays in complex environments. Users' perception and response speed to environmental hazards enhance safety.

CN118692031BActive Publication Date: 2024-11-19SUZHOU AUTOLINE TECH CO LTD
View PDF 8 Cites 0 Cited by

Patent Information

Application Number
CN202411172802.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-26
Publication Date
2024-11-19
Estimated Expiration
2044-08-26

AI Technical Summary

Technical Problem

The lack of real-time performance caused by data processing delays in complex environments affects the user's response time to environmental hazards.

Method used

Provide a potential hazard identification system based on environmental perception, including environmental information acquisition module, primary data value analysis module, sensitive subject screening module, subject audio data identification and extraction module, secondary data value analysis module and potential hazard identification module, through images and audio Continuous monitoring and analysis of data, identify and screen potential hazard subjects, and improve users' awareness and response speed to the environment.

Benefits of technology

The system can promptly identify potential hazards in the road environment, improve the safety and autonomy of people with sensory disabilities, and reduce the risk of accidents.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118692031B_ABST
    Figure CN118692031B_ABST
Patent Text Reader

Abstract

The present application provides a potential danger identification system based on environmental perception, which relates to the field of image recognition technology. The system includes: an environmental information acquisition module for obtaining environmental image sequences and environmental audio sequences; a primary data value analysis module for obtaining subject value information distribution sequences; a sensitive subject screening module for obtaining sensitive subject image sequences; a subject audio data recognition and extraction module for obtaining sensitive subject audio sequences; a secondary data value analysis module for obtaining potential subject audio sequences; and a potential danger identification module for obtaining potential danger identification results. The present application can solve the technical problem that traditional wearable devices lack real-time performance in complex environments due to data processing delays, thereby affecting the user's reaction time to environmental danger perception, and achieve the technical effect of ensuring that users can perceive and avoid potential dangers in a timely manner and improve the safety of device use through image value area recognition.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of image recognition technology, and in particular to a potential danger recognition system based on environmental perception. Background Art

[0002] Traditional potential hazard identification systems and corresponding auxiliary equipment need to capture external images in real time and process them into augmented reality or mixed reality images. This process involves the capture, transmission, processing and display of a large amount of data, which usually takes a certain amount of time. This delay may increase in complex environments, affecting the real-time performance of the equipment. For people with sensory impairments, this delay will result in insufficient response time to the perception of danger in the environment, making it difficult to avoid danger in time. For example, when a vehicle approaches suddenly or an emergency occurs, due to equipment processing delays, people with sensory impairments may not receive warning information in time, thereby increasing the risk of accidents. Summary of the invention

[0003] The purpose of this application is to provide a potential danger identification system based on environmental perception, so as to solve the technical problem that traditional wearable devices lack real-time performance due to data processing delays in complex environments, thereby affecting the user's reaction time to environmental danger perception.

[0004] In view of the above problems, the present application provides a potential danger identification system based on environmental perception.

[0005] The present application provides a potential hazard identification system based on environmental perception, the system comprising: an environmental information acquisition module, which is used to continuously perceive and collect environmental information in a road environment based on a wearable device worn by a user, and obtain an environmental image sequence and an environmental audio sequence; a one-time data value analysis module, which is used to divide and perform a one-time data value analysis on subject images in multiple environmental images in the environmental image sequence, and obtain multiple subject value information distribution sequences; a sensitive subject screening module, which is used to screen sensitive subjects according to the multiple subject value information distribution sequences, obtain multiple sensitive subjects, and obtain multiple sensitive subject image sequences of the multiple sensitive subjects by indexing in the subject image division results of the multiple environmental images, wherein the multiple sensitive subjects It includes a vehicle; a subject audio data recognition and extraction module, which is used to perform subject audio data recognition and extraction in the environmental audio sequence according to the multiple sensitive subjects, and obtain multiple sensitive subject audio sequences; a secondary data value analysis module, which is used to perform secondary data value analysis on the multiple sensitive subject images in the multiple sensitive subject image sequences respectively, obtain multiple sensitive subject image value information sequences, and screen to obtain multiple potential subject image sequences of multiple potential subjects, and index the multiple sensitive subject audio sequences to obtain multiple potential subject audio sequences; a potential hazard identification module, which is used to identify potential hazards according to the multiple potential subject image sequences and the multiple potential subject audio sequences, and output the obtained potential hazard identification results.

[0006] The technical solution provided in this application has at least the following technical effects or advantages:

[0007] The above-mentioned potential hazard identification system based on environmental perception continuously monitors and collects information about the road environment, including images and audio, and then identifies and preliminarily evaluates the value of the subjects in these images, such as vehicles, pedestrians, etc., to screen out sensitive subjects. This process is achieved by analyzing the information value of each subject in the image. Once the sensitive subjects are determined, the sequence of these subjects in the continuous images, that is, the sensitive subject image sequence, is extracted. At the same time, the sound sequence related to these sensitive subjects, such as the horn sound and brake sound of the vehicle, is identified from the audio data to form a sensitive subject audio sequence. Subsequently, these sensitive subject images are value analyzed to screen out subjects that may pose potential dangers, that is, potential subjects. At the same time, audio data related to potential subjects is also indexed from the audio sequence. Afterwards, a comprehensive analysis is performed in combination with the image and audio information of the potential subject to identify potential traffic hazards and output the hazard identification results to the user. This process helps people with sensory impairments better understand the surrounding environment and improve the safety and autonomy of travel.

[0008] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented according to the contents of the specification, and in order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are specifically cited below. It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easy to understand through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions in the present application or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0010] Figure 1 This is a schematic diagram of the structure of the potential danger identification system based on environmental perception in this application;

[0011] Figure 2 This is a flow chart of obtaining multiple subject value information distribution sequences for the potential hazard identification system based on environment perception in this application.

[0012] Description of reference numerals:

[0013] Environmental information collection module 1, primary data value analysis module 2, sensitive subject screening module 3, subject audio data recognition and extraction module 4, secondary data value analysis module 5, potential danger identification module 6. DETAILED DESCRIPTION

[0014] This application solves the technical problem that traditional wearable devices lack real-time performance in complex environments due to data processing delays, thereby affecting the user's reaction time to environmental danger perception, by providing a potential danger identification system based on environmental perception. This application achieves the technical effect of ensuring that users can perceive and avoid potential dangers in a timely manner through image value area identification, thereby improving the safety of device use.

[0015] Below, the technical solutions in the present application will be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments of the present application. It should be understood that the present application is not limited to the example embodiments described herein. Based on the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application. It should also be noted that, for the convenience of description, only the parts related to the present application are shown in the accompanying drawings, rather than all of them.

[0016] For examples, please see the attached Figure 1 , the present application provides a potential danger identification system based on environmental perception, the system specifically includes the following modules:

[0017] The environmental information acquisition module 1 is used to continuously sense and collect environmental information in the road environment based on the wearable device worn by the user, and obtain environmental image sequences and environmental audio sequences.

[0018] Specifically, in the environmental information acquisition module 1, the system terminal continuously monitors and collects image and audio data in the road environment based on the wearable device worn by the user, such as an MR helmet. This wearable device can capture visual information on the road based on the built-in camera device to form a continuous environmental image sequence, and at the same time record the surrounding audio signals, such as the vehicle's horn, accelerator and brake sounds, etc., based on the sound collection device to form an environmental audio sequence. By collecting these image and audio data, the system terminal can identify potential dangerous situations on the road, provide important safety guarantees for users, and help them travel more safely in complex traffic environments.

[0019] The primary data value analysis module 2 is used to divide the subject images in the multiple environmental images in the environmental image sequence and perform a primary data value analysis to obtain multiple subject value information distribution sequences.

[0020] Specifically, in a data value analysis module 2, the system terminal processes the collected environmental image sequence. First, the environmental image sequence is input into the pre-built subject image division channel for identification, and each subject is separated from the background to form multiple independent subject images. These subject images include different types of traffic participants such as motor vehicles, pedestrians, and motorcycles. Subsequently, the system terminal performs a data value analysis on these subject images based on the subject value analyzer to obtain multiple subject value information sets, and then organizes the multiple subject value information sets into a sequence, namely, the subject value information distribution sequence, according to the specific position of each subject image in the environmental image. This sequence not only contains the danger level information of each subject, but also retains their original order and position relationship in the image sequence, providing an important data basis for the subsequent screening of sensitive subjects and identification of potential dangers.

[0021] Further, if Figure 2 As shown, the present application provides a method of dividing a subject image in a plurality of environmental images in the environmental image sequence and performing a data value analysis to obtain a plurality of subject value information distribution sequences, including:

[0022] According to the perception data record of the road environment, a set of sample environment images is collected, and multiple subjects in the sample environment images are identified and the subject images are divided to obtain a set of sample subject image division results; using the sample environment image set and the set of sample subject image division results, based on semantic segmentation, a subject image division channel is constructed and trained, wherein the subject image division channel includes an encoder and a decoder.

[0023] In a preferred embodiment, the system terminal identifies the types of perception data contained in the real-time perception data of the road environment, and collects a series of environmental images from the sample database as a set of sample environmental images based on these perception data types. This sample environmental image set contains sample environmental images on multiple roads, and each sample environmental image contains various subjects, such as vehicles, pedestrians, cyclists, etc. Subsequently, the system terminal identifies each subject in these sample images based on historical experience and professional knowledge, and separates each subject from its background through image segmentation technology to form a separate sample subject image. After this process is completed, a set of sample subject image segmentation results will be generated, and each result in this set corresponds to a subject that has been successfully identified and segmented in the original sample subject image. After obtaining these sample environmental image sets and the corresponding sample subject image segmentation result sets, the system terminal constructs an encoder, the main function of which is to extract useful feature representations from the input image, including multiple convolutional layers, pooling layers (or downsampling layers) and activation functions. The convolutional layer applies convolution kernels to the image in a sliding window manner to extract local features. As the number of layers increases, the convolutional layer can capture more and more abstract and advanced features. The pooling layer is used to reduce the spatial dimension (i.e., height and width) of the feature map while retaining important information, reducing the amount of computation and preventing overfitting. The activation function ReLU can increase nonlinearity, allowing the model to learn complex patterns. The system terminal builds a decoder based on semantic segmentation. The decoder maps the high-level features extracted by the encoder back to the pixel space to generate an output image of the same size as the input image and realize pixel-level classification. It contains multiple upsampling layers, such as transposed convolution layers, bilinear interpolation, etc., to gradually restore the size of the feature map. During the upsampling process, the decoder combines certain layers in the encoder to supplement the detail information through jump connections to improve the accuracy of segmentation. The final convolution layer maps the feature map to the number of target categories and uses the softmax activation function to generate the probability that each pixel belongs to each category. After that, the system terminal encapsulates the encoder and decoder to form a subject image segmentation channel, and divides the sample environment image set and the sample subject image segmentation result set into a training set and a test set. Then, the sample environment image in the training set is input into the subject image segmentation channel, and the features are extracted by the internal encoder, and then the segmentation result, i.e., the category prediction of each pixel, is generated by the decoder. The loss value of the segmentation result and the corresponding sample main image segmentation result is calculated through cross entropy loss. The weight parameters of the main image segmentation channel are updated using the back propagation algorithm and the Adam optimization algorithm to minimize the loss value. The above process is repeated until the preset training rounds are reached. After the training is completed, the system terminal uses the test set to evaluate the current main image segmentation channel and calculate the segmentation accuracy, that is, the accuracy of the segmentation.If the calculated segmentation accuracy meets the preset segmentation accuracy, the system terminal outputs the current main image division channel. Otherwise, the structure, hyperparameters or the like of the main image division channel are adjusted to obtain better performance.

[0024] Based on the subject image division channel, the subject images within the multiple environmental images are divided to obtain multiple subject image sets; a data value analysis is performed on the multiple subject image sets to obtain multiple subject value information sets, and based on the position of each subject image in the environmental image, multiple subject value information distribution sequences are constructed to obtain.

[0025] In a preferred embodiment, based on the trained subject image division channel, the system terminal accurately divides the subject images of multiple environmental images in the environmental image sequence. The subject image division channel will identify different subjects in each environmental image, such as vehicles, pedestrians, etc., based on the internal encoder and decoder, and separate them from the background to form a plurality of sets containing different subject images. Subsequently, these subject image sets are input into a pre-built subject value analyzer for a data value analysis. This analysis process will evaluate the importance and potential risks of the subject in each subject image in the current road environment. For example, a vehicle approaching an intersection quickly may have a higher data value than a stationary pedestrian because it may represent a more urgent potential danger. Through analysis, the subject value analyzer assigns a corresponding value information to each subject image, which reflects the importance and possible risks of the subject in the current environment. Afterwards, based on the position information of each subject image in the original environmental image, these subject value information are organized into a sequence, namely, the subject value information distribution sequence. This sequence not only contains the value information of each subject, but also retains their spatial position relationship in the environmental image. In this way, the distribution and potential risks of different entities in the road environment can be understood more intuitively, providing strong support for subsequent identification of potential hazards.

[0026] Furthermore, the present application provides a method of performing a data value analysis on the plurality of subject image sets to obtain a plurality of subject value information sets, including:

[0027] Collect a set of sample subject images, and identify the degree of danger of the subject in each subject image to obtain a set of sample subject value information; use the sample subject image set and the sample subject value information set to train a subject value analyzer; use the subject value analyzer to perform a data value analysis on the multiple subject image sets to obtain multiple subject value information sets.

[0028] In an optional implementation, the system terminal collects a series of sample subject image sets containing different subjects from the sample database. For each sample subject image, the system terminal performs a danger level assessment based on historical experience and professional knowledge, that is, based on factors such as the subject's behavior, position, speed, etc., it determines the risk or threat that it may bring in the current environment and gives a corresponding danger level identification. After this process is completed, a sample subject value information set containing the sample subject image and its corresponding danger level identification is obtained. Subsequently, the system terminal uses a convolutional neural network (CNN) to initialize a subject value analyzer, including the number of neural network layers, the number of neurons in each layer, the size of the convolution kernel, the activation function, etc., and then uses the same training method as mentioned above to train this subject value analyzer to construct the final subject value analyzer. After that, the system terminal inputs multiple subject image sets into the subject value analyzer. For each subject image set, the subject value analyzer performs a comprehensive data value analysis based on the learned knowledge, evaluates the value of each subject therein, and generates multiple subject value information sets. These subject value information sets not only reflect the degree of danger of the subject, but also include the position of the subject in the environment.

[0029] The sensitive subject screening module 3 is used to screen sensitive subjects according to the distribution sequences of the value information of the multiple subjects, obtain multiple sensitive subjects, and index into the subject image division results of the multiple environmental images to obtain multiple sensitive subject image sequences of the multiple sensitive subjects, wherein the multiple sensitive subjects include vehicles.

[0030] Specifically, in the sensitive subject screening module 3, the system terminal distinguishes the value of the subject based on the obtained distribution sequence of multiple subject value information combined with a pre-set threshold. By comparing the value information of each subject with the threshold, the subjects that meet the threshold requirements are screened out, and these subjects are regarded as sensitive subjects. In practical applications, sensitive subjects include various types, such as vehicles, which have high potential dangers. Once the sensitive subjects are screened out, the system terminal indexes these sensitive subjects in the subject image division results of multiple environmental images and extracts the corresponding subject images. These subject images are arranged in the order in which they appear in the environmental images to form multiple sensitive subject image sequences. These image sequences provide the basis for subsequent detailed analysis and processing.

[0031] Furthermore, the present application provides a method of screening sensitive subjects according to the plurality of subject value information distribution sequences, obtaining a plurality of sensitive subjects, and obtaining a plurality of sensitive subject image sequences of the plurality of sensitive subjects by indexing in the subject image division results of the plurality of environmental images, including:

[0032] According to the distribution sequences of the value information of the multiple subjects, multiple subjects whose subject value information is greater than the value information threshold are screened as multiple sensitive subjects; within the subject image division results of the multiple environmental images, all subject images of the multiple sensitive subjects are indexed and obtained, and they are arranged in chronological order to obtain multiple sensitive subject image sequences.

[0033] In a preferred embodiment, the system terminal sets a value information threshold according to the distribution sequence of multiple subject value information. This value information threshold is determined according to business needs and expert advice, and is used to distinguish which subjects have higher value information, that is, which subjects are more sensitive or important. By comparing the size relationship between the value information of each subject and the value information threshold, the system terminal screens out those subjects whose value information is greater than the value information threshold and marks them as sensitive subjects. Subsequently, the system terminal indexes and obtains all subject images of the screened sensitive subjects in the subject image division results of multiple environmental images, and then arranges these acquired subject images in the time order in which they appear in the environmental images to form multiple sensitive subject image sequences. The purpose of this process is to maintain the temporal continuity between images, so that the sensitive subject image sequence can reflect the dynamic changes of the sensitive subject over a period of time. Each sensitive subject image sequence contains images of a sensitive subject at multiple time points, and these images are arranged in chronological order to facilitate the acquisition of subsequent sensitive subject audio sequences.

[0034] The subject audio data recognition and extraction module 4 is used to perform subject audio data recognition and extraction in the environmental audio sequence according to the multiple sensitive subjects to obtain multiple sensitive subject audio sequences.

[0035] Specifically, in the subject audio data identification and extraction module 4, the system terminal matches the audio features corresponding to the sensitive subjects in the subject audio feature library based on the determined multiple sensitive subjects, such as the specific audio frequency and decibel emitted by different vehicles or motorcycles. Subsequently, the system terminal screens and extracts the corresponding audio data from the ambient audio sequence according to these audio features, forming multiple sensitive subject audio sequences, laying the foundation for the subsequent potential danger identification.

[0036] Further, the present application provides a method of performing subject audio data recognition and extraction in the environmental audio sequence according to the multiple sensitive subjects to obtain multiple sensitive subject audio sequences, including:

[0037] The multiple sensitive subjects are input into a subject audio feature library for indexing to obtain multiple sensitive subject audio feature information, wherein the subject audio feature library includes a mapping table of subject audio feature information and subject categories; subject audio data recognition and extraction are performed in the environmental audio sequence based on the multiple sensitive subject audio feature information to obtain multiple sensitive subject audio sequences.

[0038] In a preferred embodiment, the system terminal inputs multiple sensitive subjects into the subject audio feature library for indexing. The subject audio feature library is actually a database containing the mapping relationship between the subject audio feature information and the subject category. Through this indexing process, the system terminal can quickly identify the audio feature information corresponding to each sensitive subject, for example, the unique audio frequency and timbre characteristics of the sensitive subject. The system terminal divides the audio signal in the audio feature information into multiple audio frames, each frame length is usually 20-30ms, and there is a certain overlap between frames, such as 10-15ms. A Hamming window function is applied to each frame to reduce the spectrum leakage at the edge of the frame. Subsequently, the corresponding features are extracted from each audio frame using the obtained sensitive subject audio feature information and compared. If the audio, decibel, etc. extracted from the audio frame are within the corresponding range interval in the sensitive subject audio feature information. Repeat the above process, the system terminal obtains multiple audio frames, and connects the audio frames that are adjacent in position and have audio, decibel and other features in the same interval, that is, belong to the same subject category, to form multiple sensitive subject audios. Afterwards, the system terminal aggregates the sensitive subject audios belonging to the same subject category to form multiple sensitive subject audio sequences.

[0039] The secondary data value analysis module 5 is used to perform secondary data value analysis on the multiple sensitive subject images in the multiple sensitive subject image sequences respectively, obtain multiple sensitive subject image value information sequences, and screen to obtain multiple potential subject image sequences of multiple potential subjects, and index the multiple sensitive subject audio sequences to obtain multiple potential subject audio sequences.

[0040] Specifically, in the secondary data value analysis module 5, the system terminal performs secondary data value analysis on the image sequences of multiple sensitive subjects. This step evaluates the moving speed and potential danger of the subject by comparing the similarity between adjacent images. If the similarity of adjacent images is high, it means that the subject moves slowly and the potential danger is relatively small, so the corresponding sensitive subject image value information is also small. On the contrary, if the similarity is low, it means that the subject moves quickly, the potential danger increases, and the corresponding sensitive subject image value information also increases accordingly. Through such analysis, the system terminal can generate multiple sensitive subject image value information sequences, each of which reflects the dynamic changes of the corresponding sensitive subject and its potential threat level. Subsequently, the system terminal summarizes and sorts these sensitive subject image value information sequences to identify multiple sensitive subjects with the greatest potential threat, namely potential subjects. This step is achieved by calculating the sum of the image value information of each subject and selecting the subject with the largest sum. These potential subjects represent the objects that move the fastest and have the greatest potential danger in the current environment. In order to analyze the potential danger more accurately, the system terminal selects images with a large proportion of image value information from the image value information sequence of the potential subject to form multiple potential subject image sequences. These potential subject image sequences capture key scenes when the potential subject is moving at high speed or undergoing significant changes, providing an important basis for subsequent potential hazard analysis. Afterwards, combined with the analysis results of the sensitive subject audio sequence, the system terminal confirms and locates the specific location and behavioral characteristics of the potential subject. By indexing the corresponding parts of multiple potential subjects in the audio sequence, multiple potential subject audio sequences are obtained. These sequences complement each other with the image sequence and together provide comprehensive and accurate information for the identification of potential hazards in the road environment.

[0041] Further, the present application provides performing secondary data value analysis on multiple sensitive subject images in the multiple sensitive subject image sequences respectively to obtain multiple sensitive subject image value information sequences, and screening to obtain multiple potential subject image sequences of multiple potential subjects, and indexing the multiple sensitive subject audio sequences to obtain multiple potential subject audio sequences, including:

[0042] Performing secondary data value analysis on adjacent groups of sensitive subject images within the multiple sensitive subject image sequences respectively to obtain multiple sensitive subject image value information sequences, wherein each group of sensitive subject images includes two sensitive subject images; calculating the value information based on the multiple sensitive subject image value information sequences to obtain total value information of multiple subject image sequences; and screening multiple subjects corresponding to the largest total value information of multiple subject image sequences as multiple potential subjects based on the total value information of the multiple subject image sequences.

[0043] In a preferred embodiment, the system terminal first groups two adjacent images in each sensitive subject image sequence into a group of sensitive subject images for analysis, and then performs secondary data value analysis on these adjacent image groups through a pre-built adjacent image similarity analyzer to evaluate the movement or change degree of the sensitive subject in each group of images. If the two images have a high degree of similarity, it means that the sensitive subject moves slowly or has no significant changes during this period, so its image value information is relatively low; conversely, if the similarity is low, it means that the sensitive subject moves quickly or has significant changes, and the image value information is high. In this way, the system terminal can obtain an image value information sequence that reflects the dynamic changes and value of the subject in different time periods. Subsequently, the system terminal summarizes the image value information sequences of these sensitive subjects and calculates the total value information of each subject image sequence. This total value information is a comprehensive evaluation of the dynamic changes and value of the subject during the entire observation period. Afterwards, the subject image sequence is sorted according to the total value information, and several subjects with the highest total value information are selected as potential subjects. These potential subjects are determined to be the objects with the largest dynamic changes and the highest potential threats in the current environment. Through such a screening process, the system terminal can quickly locate the subjects that need special attention and provide support for subsequent identification of potential dangers.

[0044] In the multiple sensitive subject image value information sequences of the multiple potential subjects, the sensitive subject images with the largest preset proportion of sensitive subject image value information are respectively screened to obtain multiple potential subject image sequences; according to the timestamp information of the potential subject images in the multiple potential subject image sequences, the audios of the potential subjects at the same timestamp are indexed in the multiple sensitive subject audio sequences to obtain multiple potential subject audio sequences.

[0045] In a preferred embodiment, among the multiple sensitive subject image value information sequences determined as potential subjects, the system terminal screens the sensitive subject image value information sequence of each potential subject, that is, selects the part of the image with the largest value information in the sequence, and the proportion of this part of the image is preset according to business needs and expert advice, for example, the first 10% or the first 20%, etc. The purpose of this is to focus on those images that best represent the important dynamic changes of the potential subject, thereby generating a more refined potential subject image sequence. Subsequently, the system terminal uses the timestamp information in these potential subject image sequences to associate the corresponding sensitive subject audio sequence. The timestamp is a key attribute in image and audio data, which records the exact time when the data is captured. By comparing the timestamp, the system terminal can find the audio data at the same time point as the screened potential subject image in the audio sequence. In this way, a corresponding potential subject audio sequence can be generated for each potential subject, and these audio sequences are synchronized with the image sequence in time, and together reflect the dynamic changes of the potential subject.

[0046] Furthermore, the present application provides a method of performing secondary data value analysis on adjacent groups of sensitive subject images in the plurality of sensitive subject image sequences to obtain a plurality of sensitive subject image value information sequences, including:

[0047] Combine the sensitive subject images of the previous frame and the next frame in the plurality of sensitive subject image sequences to obtain a plurality of groups of sensitive subject images; collect a set of sample sensitive subject image combinations, and analyze the similarity of two sample sensitive subject images in each sample sensitive subject image combination, and identify and obtain a sample image similarity set; construct an adjacent image similarity analyzer based on a twin network; use the sample sensitive subject image combination set and the sample image similarity set to train the adjacent image similarity analyzer until convergence;

[0048] In an optional embodiment, the system terminal combines the previous frame and the next frame of each frame image from multiple sensitive subject image sequences to form multiple groups of sensitive subject image pairs. Such a combination helps to capture the changes of sensitive subjects in continuous time. In order to train an analyzer that can analyze such changes, the system terminal collects a sample sensitive subject image combination set based on the sample database, and this set contains multiple groups of sample image pairs. For each two images in the sample image combination, the system terminal uses the feature matching algorithm FLANN to find the matching feature points between them. Subsequently, based on the matching feature points, the similarity between the two images is calculated using the Euclidean distance. The similarity value calculated is then used as an identifier of the sample image combination and stored in the sample image similarity set. This set will be used for subsequent training and evaluation processes. After obtaining the sample sensitive subject image combination set and the sample image similarity set, the system terminal constructs and trains an adjacent image similarity analyzer based on a Siamese Network. The twin network is a special neural network structure, which consists of two or more sub-networks with the same structure and shared parameters. These sub-networks receive sensitive subject images and extract features respectively, and then analyze the similarity of two adjacent frames of sensitive subject images. After that, the system terminal uses a similar training method as mentioned above to minimize the error between the similarity prediction output by the adjacent image similarity analyzer and the sample image similarity. This process continues until convergence on the validation set, that is, the loss value is in a stable state.

[0049] The adjacent image similarity analyzer is used to perform similarity analysis on the multiple groups of sensitive subject images to obtain multiple sensitive subject image similarity sequences; based on the multiple sensitive subject image similarity sequences, multiple sensitive subject image value information sequences are calculated, wherein the size of the sensitive subject image similarity is inversely correlated with the size of the sensitive subject image value information.

[0050] In an optional embodiment, the system terminal uses a trained adjacent image similarity analyzer to perform similarity analysis on multiple groups of sensitive subject images. This analyzer can accurately evaluate the similarity between two adjacent frames in each group of images and output a similarity value. By analyzing all groups of images, the system terminal obtains multiple sensitive subject image similarity sequences, each of which represents the change in similarity of the group of images over time. Subsequently, based on these sensitive subject image similarity sequences, the system terminal calculates multiple sensitive subject image value information sequences. The calculation process follows the principle that the size of the similarity is inversely correlated with the size of the value information of the sensitive subject image, that is, the similarity of the sensitive subject images is calculated in inverse proportion.

[0051] The potential danger identification module 6 is used to identify potential dangers according to the multiple potential subject image sequences and the multiple potential subject audio sequences, and output a potential danger identification result.

[0052] Specifically, in the potential danger identification module 6, the system terminal inputs multiple potential subject image sequences and multiple potential subject audio sequences into a pre-built potential danger identifier for potential danger identification, and outputs a potential danger identification result, which is a warning signal indicating the existence of potential danger.

[0053] Furthermore, the present application provides a method of performing potential hazard identification according to the multiple potential subject image sequences and the multiple potential subject audio sequences, and outputting a potential hazard identification result, including:

[0054] According to the historical data of potential danger identification, a set of sample potential subject image sequences and a set of sample potential subject audio sequences are collected, and a set of sample potential danger levels is identified and obtained; the set of sample potential subject image sequences, the set of sample potential subject audio sequences and the set of sample potential danger levels are used to train a potential danger identifier.

[0055] In a preferred embodiment, the system terminal collects a set of sample potential subject image sequences and a set of sample potential subject audio sequences from a sample database. These sample sets represent image and audio information under different potential danger scenes, respectively. In order to train the potential danger identifier, the system terminal labels each sample with a danger level to form a set of sample potential danger levels. This labeling process is carried out based on historical experience and expert advice. Subsequently, the system terminal initializes the image recognition branch according to the convolutional neural network, determines the number of neural network layers, the number of neurons in each layer, the size of the convolution kernel, the activation function, etc. Initialize the audio recognition branch according to the recurrent neural network (RNN), determine the number of RNN layers, the number of neurons in each layer, the activation function, the time step, etc. Construct a fusion layer according to the fusion mechanism, which is a weighted summation mechanism containing two weights determined based on historical experience and professional knowledge. Afterwards, the system terminal connects the image recognition branch and the audio recognition branch in parallel, and then connects them in series with the fusion layer to form a potential danger identifier. Then, the system terminal uses a similar training method as described above to update the weight parameters of the potential danger identifier through the back propagation algorithm and the optimization algorithm to minimize the loss function value.

[0056] The potential danger identifier is used to perform potential danger identification on the multiple potential subject image sequences and the multiple potential subject audio sequences respectively, to obtain multiple potential danger levels, and to calculate a comprehensive potential danger level; it is determined whether the comprehensive potential danger level is greater than a potential danger level threshold to obtain a potential danger identification result.

[0057] In a preferred embodiment, the system terminal uses the image recognition branch and the audio recognition branch in the trained potential danger identifier to independently analyze and process the input multiple potential subject image sequences and audio sequences. The two branches independently identify the potential dangers, give the corresponding potential danger levels, and perform weighted summation through the fusion layer to generate the final potential danger level. Repeat the above process, and the system terminal obtains multiple potential danger levels. Subsequently, these potential danger levels are comprehensively calculated to obtain a comprehensive potential danger level. The comprehensive calculation involves calculating the mean of multiple potential danger levels. After that, the calculated comprehensive potential danger level is compared with the preset potential danger level threshold. This threshold is set according to business needs and expert advice, and is used to determine whether the potential danger in the current situation has reached the level that requires action. If the comprehensive potential danger level is greater than the threshold, it indicates that there is a significant potential danger, and the system terminal will output the corresponding potential danger identification result so that timely countermeasures can be taken.

[0058] In summary, the potential danger identification system based on environmental perception provided by this application has the following technical effects:

[0059] The present application continuously senses and collects environmental information in the road environment through the environmental information acquisition module 1 to obtain an environmental image sequence and an environmental audio sequence. The primary data value analysis module 2 divides and analyzes the subject images in multiple environmental images in the environmental image sequence to obtain a subject value information distribution sequence. The sensitive subject screening module 3 screens out sensitive subjects, such as vehicles, based on these distribution sequences, and indexes the image sequences of these sensitive subjects in the image. The subject audio data identification and extraction module 4 identifies and extracts the corresponding audio data in the audio sequence based on these sensitive subjects to obtain a sensitive subject audio sequence. The secondary data value analysis module 5 performs a secondary analysis on the sensitive subject image sequence, screens out the image sequence of potential subjects, and indexes the corresponding audio data in the audio sequence. The potential danger identification module 6 identifies potential dangers based on these image and audio sequences and outputs the danger identification results. These technical effects jointly solve the technical problem that the traditional wearable device is not real-time enough due to data processing delays in complex environments, thereby affecting the user's reaction time to environmental danger perception, and achieves the technical effect of ensuring that users can perceive and avoid potential dangers in a timely manner and improve the safety of device use through image value area recognition.

[0060] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

[0061] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application belong to the scope of the present application and its equivalent technology, the present application is also intended to include these modifications and variations.

Claims

1. Potential danger identification system based on environmental perception, characterized by: The system comprises: The environmental information acquisition module is used to continuously sense and collect environmental information in the road environment based on the wearable device worn by the user, and obtain environmental image sequences and environmental audio sequences; A primary data value analysis module, used to divide the subject images in the multiple environmental images in the environmental image sequence and perform a primary data value analysis to obtain multiple subject value information distribution sequences; A sensitive subject screening module, configured to screen sensitive subjects according to the plurality of subject value information distribution sequences, obtain a plurality of sensitive subjects, and obtain a plurality of sensitive subject image sequences of the plurality of sensitive subjects by indexing in the subject image division results of the plurality of environment images, wherein the plurality of sensitive subjects include vehicles; A subject audio data recognition and extraction module, used to perform subject audio data recognition and extraction in the environmental audio sequence according to the multiple sensitive subjects, to obtain multiple sensitive subject audio sequences; A secondary data value analysis module is used to perform secondary data value analysis on multiple sensitive subject images in the multiple sensitive subject image sequences respectively to obtain multiple sensitive subject image value information sequences, and to screen and obtain multiple potential subject image sequences of multiple potential subjects, and to index the multiple sensitive subject audio sequences to obtain multiple potential subject audio sequences; A potential danger identification module, used to identify potential dangers according to the multiple potential subject image sequences and the multiple potential subject audio sequences, and output a potential danger identification result; The subject images in the plurality of environmental images in the environmental image sequence are divided and subjected to a data value analysis to obtain a plurality of subject value information distribution sequences, including: According to the perception data record of the road environment, a set of sample environment images is collected, and multiple subjects in the sample environment images are identified and the subject images are divided to obtain a set of sample subject image division results; Using the sample environment image set and the sample subject image segmentation result set, based on semantic segmentation, construct and train a subject image segmentation channel, wherein the subject image segmentation channel includes an encoder and a decoder; Based on the subject image division channel, the subject image in the multiple environment images is divided to obtain multiple subject image sets; Performing a data value analysis on the plurality of subject image sets to obtain a plurality of subject value information sets, and constructing a plurality of subject value information distribution sequences based on the position of each subject image in the environment image; According to the plurality of subject value information distribution sequences, sensitive subject screening is performed to obtain a plurality of sensitive subjects, and a plurality of sensitive subject image sequences of the plurality of sensitive subjects are obtained by indexing in the subject image division results of the plurality of environmental images, including: According to the distribution sequences of the value information of the multiple subjects, multiple subjects whose subject value information is greater than a value information threshold are screened as multiple sensitive subjects; In the subject image division results of the multiple environmental images, all subject images of the multiple sensitive subjects are obtained by indexing, and arranged in chronological order to obtain multiple sensitive subject image sequences; Performing secondary data value analysis on multiple sensitive subject images in the multiple sensitive subject image sequences respectively to obtain multiple sensitive subject image value information sequences, screening to obtain multiple potential subject image sequences of multiple potential subjects, and indexing the multiple sensitive subject audio sequences to obtain multiple potential subject audio sequences, including: Performing secondary data value analysis on adjacent multiple groups of sensitive subject images in the multiple sensitive subject image sequences respectively to obtain multiple sensitive subject image value information sequences, wherein each group of sensitive subject images includes two sensitive subject images; Calculate the value information based on the multiple sensitive subject image value information sequences to obtain the total value information of the multiple subject image sequences; According to the total value information of the plurality of subject image sequences, selecting a plurality of subjects corresponding to the maximum total value information of the plurality of subject image sequences as a plurality of potential subjects; In the multiple sensitive subject image value information sequences of the multiple potential subjects, respectively select sensitive subject images with a preset ratio of the largest sensitive subject image value information to obtain multiple potential subject image sequences; According to the timestamp information of the potential subject images in the multiple potential subject image sequences, the audio of the potential subject at the same timestamp is indexed in the multiple sensitive subject audio sequences to obtain multiple potential subject audio sequences; Performing secondary data value analysis on adjacent groups of sensitive subject images in the plurality of sensitive subject image sequences respectively to obtain a plurality of sensitive subject image value information sequences, including: Combining the sensitive subject images of the previous frame and the next frame in the plurality of sensitive subject image sequences to obtain a plurality of groups of sensitive subject images; Collecting a set of sample sensitive subject image combinations, and analyzing the similarity between two sample sensitive subject images in each sample sensitive subject image combination, and identifying a sample image similarity set; Based on the twin network, a similarity analyzer for adjacent images is constructed; Using the sample sensitive subject image combination set and the sample image similarity set, training the adjacent image similarity analyzer until convergence; Using the adjacent image similarity analyzer, similarity analysis is performed on the multiple groups of sensitive subject images to obtain multiple sensitive subject image similarity sequences; Based on the multiple sensitive subject image similarity sequences, multiple sensitive subject image value information sequences are calculated and obtained, wherein the magnitude of the sensitive subject image similarity is inversely correlated with the magnitude of the sensitive subject image value information.

2. The potential danger identification system based on environmental perception according to claim 1 is characterized in that: Performing a data value analysis on the plurality of subject image sets to obtain a plurality of subject value information sets, including: Collect a set of sample subject images, and identify the degree of danger of the subject in each subject image to obtain a set of sample subject value information; Using the sample subject image set and the sample subject value information set to train a subject value analyzer; The subject value analyzer is used to perform a data value analysis on the multiple subject image sets to obtain multiple subject value information sets.

3. The potential danger identification system based on environmental perception according to claim 1 is characterized in that: According to the multiple sensitive subjects, subject audio data recognition and extraction are performed in the environmental audio sequence to obtain multiple sensitive subject audio sequences, including: Input the multiple sensitive subjects into a subject audio feature library for indexing to obtain multiple sensitive subject audio feature information, wherein the subject audio feature library includes a mapping table between subject audio feature information and subject categories; Based on the multiple sensitive subject audio feature information, subject audio data recognition and extraction are performed in the environmental audio sequence to obtain multiple sensitive subject audio sequences.

4. The potential danger identification system based on environmental perception according to claim 1 is characterized in that: According to the multiple potential subject image sequences and the multiple potential subject audio sequences, potential danger identification is performed, and a potential danger identification result is output, including: According to the potential danger identification historical data, a sample potential subject image sequence set and a sample potential subject audio sequence set are collected, and a sample potential danger level set is identified and obtained; Using the sample potential subject image sequence set, the sample potential subject audio sequence set and the sample potential danger level set to train a potential danger identifier; Using the potential danger identifier, respectively identify the potential dangers of the multiple potential subject image sequences and the multiple potential subject audio sequences to obtain multiple potential danger levels, and calculate and obtain a comprehensive potential danger level; It is determined whether the comprehensive potential danger level is greater than the potential danger level threshold to obtain a potential danger identification result.

Citation Information

Patent Citations

  • Pre-warning method and device based on wearable device

    CN105632049A

  • System and method for external sound synthesis of a vehicle

    CN107018467A

  • Target behavior recognition and prediction method in complex dynamic environment

    CN108647582A

  • Video monitoring method and device, computer equipment and storage medium

    CN110620905A

  • Public safety risk early warning system and method

    CN113222221A