A traffic jam state determination method and device, vehicle and storage medium

Abstract

The application discloses a traffic jam state determination method and device, a vehicle and a storage medium. The method comprises the following steps: when it is monitored that the driving speed of the vehicle meets the congestion early warning vehicle speed, acquiring continuous frame information captured by a camera device of the vehicle and an environmental audio signal collected by an audio device of the vehicle; determining a first motion state and a second motion state of a preceding vehicle according to the continuous frame information, and determining preceding vehicle sound information according to the environmental audio signal, wherein the preceding vehicle comprises at least one vehicle in front of the vehicle; determining a preceding vehicle motion state of the preceding vehicle according to the first motion state and the second motion state, and determining a preceding vehicle sound state of the preceding vehicle according to the preceding vehicle sound information; and determining and displaying a traffic jam state according to the preceding vehicle motion state and the preceding vehicle sound state. The method solves the problem of large judgment error caused by visual error or perception difference of the driver, quickly and accurately provides the current traffic jam state for the driver, and improves the safety and experience of the driver driving the vehicle.

Description

Technical Field

[0001] This invention relates to the field of vehicle technology, and more particularly to a method, apparatus, vehicle, and storage medium for determining traffic congestion status. Background Technology

[0002] In today's driving environment, drivers frequently encounter traffic congestion, such as highway traffic jams during holidays. In such situations, drivers typically rely on visual observation or subjective judgment to determine whether they are still in a traffic jam. If the traffic jam ends, they control the vehicle to start moving and accelerate away.

[0003] However, the above methods for judging traffic congestion, if relying solely on the driver's observation or subjective judgment of the traffic congestion status, will produce significant judgment errors due to the driver's visual errors or perceptual differences. Summary of the Invention

[0004] This invention provides a method, device, vehicle, and storage medium for determining traffic congestion status, which solves the problem of large judgment errors caused by the driver's visual errors or perceptual differences, and provides the driver with the current traffic congestion status quickly and accurately, thereby improving the driver's safety and driving experience.

[0005] According to one aspect of the present invention, a method for determining traffic congestion status is provided, applied to a vehicle; the method includes:

[0006] When the vehicle's speed is detected to meet the congestion warning speed, the system acquires continuous frame information captured by the vehicle's camera equipment and environmental audio signals collected by the vehicle's audio equipment.

[0007] The first motion state and the second motion state of the vehicle in front are determined based on continuous frame information, and the sound information of the vehicle in front is determined based on the ambient audio signal, wherein the vehicle in front includes at least one vehicle in front of the vehicle.

[0008] The motion state of the preceding vehicle is determined based on the first motion state and the second motion state, and the sound state of the preceding vehicle is determined based on the sound information of the preceding vehicle.

[0009] The traffic jam status is determined and displayed based on the movement and sound of the vehicle in front.

[0010] The traffic congestion determination method provided in this invention, when detecting that the vehicle's speed meets the congestion warning speed, acquires continuous frame information captured by the vehicle's camera and environmental audio signals collected by the vehicle's audio equipment; determines the first and second motion states of the preceding vehicle based on the continuous frame information, and determines the preceding vehicle's sound information based on the environmental audio signals; determines the preceding vehicle's motion state based on the first and second motion states, and determines the preceding vehicle's sound state based on the preceding vehicle's sound information; and determines and displays the traffic congestion state based on the preceding vehicle's motion and sound states. This technical solution, on the one hand, controls the camera and audio equipment to start collecting data only when the vehicle's speed meets the congestion warning speed, which can reduce vehicle energy consumption and provide accurate data support for subsequent traffic congestion determination. On the other hand, the first and second motion states are presented in a visual form, while the preceding vehicle's sound information is presented in an auditory form. Therefore, obtaining the preceding vehicle's motion and sound states can provide diverse common judgment indicators for subsequent traffic congestion determination from both visual and auditory perspectives. Finally, based on the visual modality of the preceding vehicle's motion state and the auditory modality of the preceding vehicle's sound state, the traffic jam status is jointly determined. This solves the problem that existing technologies may produce large judgment errors due to the driver's visual errors or perceptual differences, providing the driver with the current traffic jam status quickly and accurately, thus improving the driver's safety and driving experience.

[0011] According to another aspect of the present invention, a traffic jam state determination device is provided, applied to a vehicle; the device includes:

[0012] The acquisition module is used to acquire continuous frame information captured by the vehicle's camera equipment and environmental audio signals collected by the vehicle's audio equipment when the vehicle's driving speed is detected to meet the congestion warning speed.

[0013] The processing module is used to determine the first motion state and the second motion state of the vehicle in front based on continuous frame information, and to determine the sound information of the vehicle in front based on the ambient audio signal, wherein the vehicle in front includes at least one vehicle in front of the vehicle.

[0014] The determination module is used to determine the motion state of the preceding vehicle based on the first motion state and the second motion state, and to determine the sound state of the preceding vehicle based on the sound information of the preceding vehicle.

[0015] The display module is used to determine and display the traffic jam status based on the movement and sound of the vehicle in front.

[0016] According to another aspect of the present invention, a vehicle is provided, the vehicle comprising:

[0017] Electronic devices;

[0018] The electronic device includes at least one processor; and

[0019] A memory communicatively connected to the at least one processor; wherein,

[0020] The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the traffic congestion status determination method according to any embodiment of the present invention.

[0021] According to another aspect of the present invention, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the traffic jam state determination method of any embodiment of the present invention.

[0022] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the traffic jam state determination method according to any embodiment of the present invention.

[0023] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 A flowchart illustrating a method for determining traffic congestion status according to an embodiment of the present invention;

[0026] Figure 2 A flowchart illustrating another method for determining traffic congestion status provided in an embodiment of the present invention;

[0027] Figure 3 This is a schematic diagram of a traffic jam status determination device provided in an embodiment of the present invention;

[0028] Figure 4 This is a schematic diagram of the structure of an electronic device in a vehicle provided in an embodiment of the present invention. Detailed Implementation

[0029] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0031] Figure 1 This is a flowchart illustrating a traffic congestion determination method according to an embodiment of the present invention. This embodiment is applicable to situations where a driver encounters road congestion while driving. The method can be executed by a traffic congestion determination device, which can be implemented in hardware and / or software and can be configured in the vehicle's electronic equipment. In this embodiment, the electronic equipment can be a vehicle controller, engine control unit, or hybrid power controller, specifically determined according to the vehicle type. Figure 1 As shown, the method includes:

[0032] S101. When the vehicle's speed is detected to meet the congestion warning speed, acquire continuous frame information captured by the vehicle's camera equipment and environmental audio signals collected by the vehicle's audio equipment.

[0033] The congestion warning speed is the vehicle speed threshold when road congestion occurs. In this embodiment, the congestion warning speed can be obtained through experiments or tests, or it can be determined based on the current city, road type, or speed limit of the current road. Continuous frame information refers to image frames captured by the camera device and arranged sequentially in time. The ambient audio signal refers to the ambient sound signals around the vehicle collected by the audio device.

[0034] Specifically, vehicles can determine their congestion warning speed based on the road type and the speed limit of the current road. Furthermore, when the vehicle's current speed is detected to meet the congestion warning speed, the camera installed in front of the vehicle and the audio equipment installed around the vehicle are activated. The camera captures images of the area in front of the vehicle, obtaining continuous frame information; the audio equipment captures sounds from the surrounding area, obtaining environmental audio signals.

[0035] For example, according to an adaptive system for an elevated highway ramp, when the vehicle speed on the bridge is below 20 km / h, it is considered severely congested; between 20-35 km / h, it may be moderately congested; and between 35-50 km / h, it may be lightly congested. In this case, the congestion warning speed can be determined to be 0-50 km / h. When the vehicle speed is between 0-50 km / h, the vehicle's speed meets the congestion warning speed. At this time, the camera and audio equipment will be activated.

[0036] In this embodiment, the camera and audio equipment will only start collecting data when the vehicle's speed meets the congestion warning speed. This can reduce the vehicle's energy consumption and provide accurate data support for determining the traffic jam status later.

[0037] S102. Determine the first motion state and the second motion state of the vehicle in front based on the continuous frame information, and determine the sound information of the vehicle in front based on the ambient audio signal.

[0038] In this embodiment, the first motion state and the second motion state are the motion states of the vehicle in front, obtained from observing the actions of the vehicle in front from different angles. The first motion state is the change in the position of the vehicle in front within a series of frames, and the second motion state is the change in the vehicle indicator lights of the vehicle in front within the series of frames. The vehicle in front includes at least one vehicle directly in front of the vehicle.

[0039] Specifically, by analyzing the video captured by the camera in consecutive frames, the displacement changes of the vehicle in front can be obtained in each frame, i.e., the first motion state. Furthermore, the changes in the indicator lights of the vehicle in front can also be obtained, i.e., the second motion state. Additionally, based on ambient audio signals, the sounds emitted by the vehicle in front or other vehicles in the environment can be determined.

[0040] S103. Determine the motion state of the preceding vehicle based on the first motion state and the second motion state, and determine the sound state of the preceding vehicle based on the sound information of the preceding vehicle.

[0041] Specifically, the first motion state can determine whether the preceding vehicle has shifted, and the second motion state can predict the possible motion triggered by the preceding vehicle. Therefore, the motion state of the preceding vehicle can be determined based on the first and second motion states. In this embodiment, there can be multiple preceding vehicles, thus the motion states of multiple preceding vehicles can be determined. Furthermore, the presence of sounds that would occur when a vehicle starts or accelerates can be determined from the preceding vehicle's sound information, thereby judging the state of the preceding vehicle.

[0042] In this embodiment, the first motion state and the second motion state are presented in a visual form to show the state of the vehicle in front, while the sound information of the vehicle in front is presented in the form of sound. Therefore, obtaining the motion state and sound state of the vehicle in front can provide diverse common judgment indicators for subsequent determination of traffic jam status from both visual and sound perspectives.

[0043] S104. Determine and display the traffic jam status based on the movement and sound of the vehicle in front.

[0044] Specifically, the motion status of the vehicle in front reflects its visual modality, while the sound status reflects its auditory modality. By combining the visual modality of the vehicle's motion status and the auditory modality of its sound status, a joint judgment of the traffic jam status can be made based on different modalities to determine the current traffic jam condition. Furthermore, the traffic jam status can be displayed to the driver.

[0045] In this embodiment, the traffic jam status is jointly determined based on the visual modality of the preceding vehicle's motion state and the auditory modality of the preceding vehicle's sound state. This solves the problem that existing technologies may produce large judgment errors due to the driver's visual errors or perceptual differences, providing the driver with the current traffic jam status quickly and accurately, thereby improving the driver's driving safety and experience.

[0046] The traffic congestion determination method provided in this invention, when detecting that the vehicle's speed meets the congestion warning speed, acquires continuous frame information captured by the vehicle's camera and environmental audio signals collected by the vehicle's audio equipment; determines the first and second motion states of the preceding vehicle based on the continuous frame information, and determines the preceding vehicle's sound information based on the environmental audio signals; determines the preceding vehicle's motion state based on the first and second motion states, and determines the preceding vehicle's sound state based on the preceding vehicle's sound information; and determines and displays the traffic congestion state based on the preceding vehicle's motion and sound states. This technical solution, on the one hand, controls the camera and audio equipment to start collecting data only when the vehicle's speed meets the congestion warning speed, which can reduce vehicle energy consumption and provide accurate data support for subsequent traffic congestion determination. On the other hand, the first and second motion states are presented in a visual form, while the preceding vehicle's sound information is presented in an auditory form. Therefore, obtaining the preceding vehicle's motion and sound states can provide diverse common judgment indicators for subsequent traffic congestion determination from both visual and auditory perspectives. Finally, based on the visual modality of the preceding vehicle's motion state and the auditory modality of the preceding vehicle's sound state, the traffic jam status is jointly determined. This solves the problem that existing technologies may produce large judgment errors due to the driver's visual errors or perceptual differences, providing the driver with the current traffic jam status quickly and accurately, thus improving the driver's safety and driving experience.

[0047] Figure 2 This is a flowchart illustrating another method for determining traffic congestion status provided by an embodiment of the present invention. Based on the above embodiments, this embodiment provides a detailed description of the steps following the determination of the first and second motion states of the preceding vehicle, the determination of the preceding vehicle's motion state based on the first and second motion states, and the determination and display of the traffic congestion status. Figure 2 As shown, the method includes:

[0048] S201. When the vehicle's speed is detected to meet the congestion warning speed, acquire continuous frame information captured by the vehicle's camera equipment and environmental audio signals collected by the vehicle's audio equipment.

[0049] For example, if the vehicle is traveling on a highway with a known speed limit of 60-120 km / h, then 30 km / h can be used as the maximum speed limit for congestion warning, meaning the congestion warning speed is 0-30 km / h. Since the vehicle's speed is within the 0-30 km / h range, the congestion warning speed is met, allowing the camera to capture continuous frames and the audio equipment to collect ambient audio signals.

[0050] S202. Calculate the instantaneous velocity vector of the pixel based on the degree of pixel intensity change between each two adjacent frames in the continuous frame information.

[0051] Specifically, since the pixel brightness changes of the same object in two adjacent frames are basically the same, the pixel intensity changes between adjacent frames can be calculated to determine whether the same object has changed. Therefore, the instantaneous velocity vector of each pixel can be obtained by subtracting the pixel intensity of the previous frame from the next frame.

[0052] S203. Determine traffic flow information based on the vector direction and vector components of the instantaneous velocity vector.

[0053] Specifically, the instantaneous velocity vector includes two parameters: horizontal velocity and vertical velocity, which are called vector components. Furthermore, the vector direction can be determined based on the direction of the resultant of these two parameters. Therefore, the average velocity and direction can be calculated based on the vector direction and vector components. The calculated average velocity and direction can reflect the dominant direction and velocity of the overall flow, and thus determine traffic flow information.

[0054] Alternatively, S202 and S203 can be replaced by deep learning optical flow, the Horn-Schunck algorithm, or the Lucas-Kanade algorithm.

[0055] It is worth noting that after S202-S203 are executed, S209 can be executed together with S204-S208.

[0056] In this embodiment, by comparing the pixel intensity of each two adjacent frames in continuous frame information, information that can reflect the overall traffic flow can be determined, which can accurately determine the current overall traffic flow and provide a basis for determining the traffic congestion status.

[0057] S204. Preprocess the continuous frame information and the obtained environmental audio signal.

[0058] Specifically, for consecutive frame information, preprocessing methods include image denoising, dynamic exposure correction, and illumination equalization to enhance image quality in harsh environments such as low light, strong light, or rain and snow. These preprocessing methods can also be applied to consecutive frames in S202-S203. Furthermore, for ambient audio signals, preprocessing methods include noise reduction, echo suppression, and bandpass filtering (e.g., 300Hz-3000Hz) to improve audio signal clarity and filter out background noise.

[0059] In this embodiment, preprocessing of continuous frame information, especially processing of image light intensity, not only denoises the image and ensures the accuracy of continuous frame information, but also improves the accuracy of continuous frame information under poor weather conditions. Furthermore, preprocessing of environmental audio signals not only denoises the audio signals and improves their clarity, but also pre-retains audio in the desired frequency bands and filters out some unwanted frequency bands, ensuring the accuracy of the audio signal.

[0060] S205. Perform target recognition on the preprocessed continuous frame information to identify at least one vehicle in front, and extract features from the preprocessed environmental audio signal to obtain the sound of the vehicle in front.

[0061] Among them, the sound of the vehicle in front is a preliminary sound feature that can reflect the status of the vehicle in front after feature extraction of the environmental audio signal.

[0062] Specifically, using target recognition, at least one vehicle in front is identified from the preprocessed consecutive frames. For example, the first vehicle in front of the current vehicle, or even the second vehicle in front of the current vehicle, etc. Additionally, feature extraction is performed on the environmental audio signal to extract audio sounds with the desired features, which are then used as the status sounds of the vehicle in front.

[0063] Optionally, target recognition can be performed using target detection and tracking methods, such as target detection models based on YOLO detection models, Single-Shot Multiple-Box Detectors (SSD), or deep learning structures with self-attention mechanisms. Furthermore, the environmental audio signal can be first segmented into frames, and each frame can be windowed using Short-Time Fourier Transform (STFT) followed by Fast Fourier Transform to convert it from the time domain to the frequency domain. Then, Mel-frequency cepstral coefficients (MFCC) are used to extract features to obtain the sound of the vehicle ahead. Finally, the obtained sound of the vehicle ahead can be normalized, differentially analyzed, and fused.

[0064] Optionally, in this embodiment, target tracking can also be performed using Kalman filtering or the SORT sorting algorithm. Furthermore, semantic segmentation can be performed on continuous frame information to identify elements such as lane lines, vehicles, pedestrians, and traffic lights, further determining whether the lane ahead has been cleared.

[0065] S206. For each preceding vehicle, determine a first motion state based on the pixel information of the preceding vehicle in the continuous frame information; and determine a second motion state based on the indicator lights of the preceding vehicle.

[0066] Specifically, for each preceding vehicle, the first motion state can be obtained by determining whether the preceding vehicle has moved based on its pixel information in consecutive frames. Furthermore, the preceding vehicle's indicator lights can be used to predict its next possible action, i.e., the second motion state.

[0067] For example, pixel information includes pixel range and pixel position; determining the first motion state includes:

[0068] (i) For each frame of information in the continuous frame information, determine the pixel range and pixel position of the preceding vehicle in that frame information.

[0069] Specifically, after selecting a vehicle as the target, it can be determined whether the vehicle has moved, and the distance it has moved after moving, based on continuous frame information. For example, the pixel range occupied by the vehicle in the first frame can be obtained, and its pixel position within the entire first frame can be determined. Furthermore, the pixel range and pixel position of the vehicle in the second, third, and subsequent frames can be obtained.

[0070] (ii) Determine the first motion state based on the pixel range and pixel position of the preceding vehicle in each frame of information.

[0071] Specifically, it can be determined whether the range of the preceding vehicle has changed based on the pixel range of the preceding vehicle in the previous frame and the pixel range of the preceding vehicle in the next frame, and it can be determined whether the position of the preceding vehicle has changed based on the pixel position of the preceding vehicle in the previous frame and the pixel position of the preceding vehicle in the next frame.

[0072] For example, firstly, the range of the preceding vehicle within each frame's pixels is selected to obtain the pixel range. Based on the pixel range, the coordinates of the vehicle's center point can be determined. Then, based on the center point coordinates of the preceding vehicle in the preceding and following frames, the lateral and longitudinal displacement vectors of the preceding vehicle can be calculated. Furthermore, some feature points of the preceding vehicle, such as its edges, corners, and locations with significant texture changes, can be used as feature points, and their pixel positions can be determined. Based on the pixel positions of the feature points, the average displacement is determined. Finally, the degree of change in the pixel range of the preceding vehicle in two adjacent frames is determined, and the average displacement is determined based on the pixel positions of the feature points of the preceding vehicle in two adjacent frames. When the average displacement exceeds a certain threshold, it is determined that the preceding vehicle has moved. Based on the degree of change in the pixel range and the calculated displacement of the preceding vehicle, it can be determined between which frames the preceding vehicle moved, and based on the obtained average displacement, the actual object distance, and the focal length of the camera device, the first motion state of the preceding vehicle can be calculated.

[0073] Specifically, determining the second motion state based on the indicator lights of the vehicle ahead can be as follows:

[0074] Typical vehicles have indicator lights such as left turn signals, right turn signals, brake lights, and start lights. Therefore, the motion state of the vehicle in front can be predicted based on the indicator lights of the vehicle in front captured in each frame of a continuous series of frames, as well as the changes in the indicator lights between adjacent frames.

[0075] For example, if a vehicle in front activates its left or right turn signal in a given frame, it can be predicted that the vehicle is preparing to change lanes, thus indicating that congestion in the lane the vehicle is about to change into is ending. Similarly, based on the brake lights of a vehicle in front going from on to off in several adjacent frames, it can be determined that the vehicle is preparing to start moving, thus indicating that lane congestion is about to end. Therefore, motion prediction information, i.e., a second motion state, can be provided to the vehicle based on the indicator lights of the vehicle in front. Thus, detecting changes in the indicator lights of the vehicle in front can serve as an auxiliary method for determining the vehicle's intention to start moving.

[0076] In this embodiment, the first motion state can be determined based on the position change of the preceding vehicle in the continuous frame information, and the second motion state can be determined based on the indicator lights of the preceding vehicle. This realizes that based on the continuous frame information of the preceding vehicle, the traffic jam state can be accurately and diversely determined from the visualized information according to the situation of the preceding vehicle.

[0077] S207. Input the preceding vehicle's status sound into a pre-trained preceding vehicle sound event classifier, determine the preceding vehicle sound event, and determine the preceding vehicle sound information based on the temporal characteristics of the preceding vehicle sound event.

[0078] The original model of the sound event classifier is an end-to-end deep neural network model. After training with labeled audio samples, the sound event classifier can identify key traffic jam ending sounds after inputting audio, such as: the sound of the engine starting of the car in front (a low-frequency vibration sound), the sound of tire friction when starting, the sound of airflow enhancement during vehicle acceleration (white noise will be amplified after the car starts), and continuous horn sounds, prompts, and other sounds reminding people that they can start.

[0079] Specifically, by inputting the sound of the preceding vehicle into the preceding vehicle sound event classifier, one or more preceding vehicle sound events can be obtained. At this point, the preceding vehicle sound information can be determined based on the temporal characteristics of the preceding vehicle sound events, such as the duration of the sound occurrence, frequency changes, and sound signal strength.

[0080] For example, determining the sound information of the vehicle ahead includes:

[0081] (i) Obtain the first weighted index corresponding to the sound event of the vehicle ahead.

[0082] The first weighting index is the weight value corresponding to the different time-domain characteristics of different preceding vehicle sound events.

[0083] Specifically, a corresponding weight index can be pre-set for each sound event from the vehicle ahead. Alternatively, a primary weight index can be assigned to each sound based on its distance, intensity, frequency of occurrence, and credibility.

[0084] (ii) Determine the sound information of the vehicle in front based on the continuous sound event, frequency change, amplitude change, and the first weight index corresponding to the sound event of the vehicle in front.

[0085] Among them, the time-domain features include the duration of sound, frequency variation, and amplitude variation.

[0086] Specifically, each sound event, its duration, frequency variation, amplitude variation, and corresponding first weighting index can be used to determine the sound information of the vehicle ahead. Therefore, when multiple sound events are acquired, multiple sets of sound information of the vehicle ahead can be determined.

[0087] In this embodiment, based on the sound events of the preceding vehicle and their corresponding time-domain features, such as frequency changes, amplitude changes, etc., and their corresponding first weight index, each sound event can be grouped into a sound information of the preceding vehicle, realizing the integration of information of each sound event, which facilitates the subsequent determination of the sound status of the preceding vehicle.

[0088] S208. Based on the first motion state, the second motion state, and traffic flow information, determine the motion state of the vehicle in front, and determine the sound state of the vehicle in front based on the sound information of the vehicle in front.

[0089] Specifically, based on the first motion state, the second motion state, and traffic flow information, the motion state of the vehicle ahead is determined, including:

[0090] (i) Determine the vehicle motion state for the first motion state, the second motion state, and the traffic flow information respectively, and obtain the first state corresponding to the first motion state, the second state corresponding to the second motion state, and the third state corresponding to the traffic flow information.

[0091] The first, second, and third states can each be one of the following three states: the vehicle is in motion, the vehicle is in a congested state, or the vehicle is in a semi-congested state. The semi-congested state is characterized by vehicles moving intermittently, meaning that although stuck in traffic, the vehicles can move forward little by little.

[0092] Specifically, the first motion state is the displacement information of the vehicle in front. Therefore, upon obtaining the displacement information of the vehicle in front, the corresponding first state can be determined. For example, if the displacement information of the vehicle in front is greater than a preset displacement, the first state is determined to be that the vehicle is in motion; if the displacement information is less than or equal to the preset displacement, or the displacement information is less than a minimum displacement threshold, the first state is determined to be that the vehicle is in a congested state. Similarly, the second motion state is the motion prediction information. If the motion prediction information is that the brake lights of the vehicle in front are off, the second state can be determined to be that the vehicle is in motion; if the motion prediction information is that the brake lights of the vehicle in front are on, the second state is that the vehicle is in a congested state. Traffic flow information can reflect the overall traffic state. Therefore, if the traffic flow information indicates that traffic is in a flowing state, the third state can be determined to be that the vehicle is in motion; if the traffic flow information indicates that traffic is in a non-flowing state, the third state is that the vehicle is in a congested state.

[0093] (ii) Determine whether the first state, the second state, and the third state are all the same; if yes, then execute (iii); if no, then execute (iv).

[0094] Specifically, it needs to be determined whether these three states are the same. If all three states are the same, for example, the vehicle is in motion, then (iii) can be executed. If any one of the three states is different from the others, then (iv) needs to be executed.

[0095] (iii) If the first state, the second state and the third state are all the same, then the consistent state is directly determined as the state of motion of the vehicle in front.

[0096] Specifically, if all three states are the same, then that state can be directly taken as the state of motion of the vehicle in front. For example, if all three states indicate that the vehicle is in motion, then the state of motion of the vehicle in front is the state of motion of the vehicle in front.

[0097] (iv) If at least one of the first state, the second state and the third state is different, the dynamic weight of the first motion state, the dynamic weight of the second motion state and the dynamic weight of traffic flow information shall be determined based on the number of identical states, the current time, the time when the congestion warning speed is met and the second weight index; the motion state of the preceding vehicle shall be determined from the first state, the second state and the third state based on the dynamic weight of the first motion state, the dynamic weight of the second motion state and the dynamic weight of traffic flow information.

[0098] The second weighting indicator includes pre-set base weights for the first motion state, the second motion state, and traffic flow information. The current time is the time when each state is obtained.

[0099] Specifically, if at least one of the first, second, and third states is different (i.e., two states are the same and the other state is different from both states), or if none of the three states are the same, then the dynamic weight of the first motion state, the dynamic weight of the second motion state, and the dynamic weight of traffic flow information can be determined based on the number of identical states, the current time, the time when the congestion warning speed is met, and the second weight index.

[0100] For example, if the first state is a semi-congested state, the second state is a moving state, and the third state is a congested state, then all three states are different. Conversely, if the first state is a moving state, the second state is a semi-congested state, and the third state is a moving state, then two of these states are the same. In this case, based on the current time of each state and the time required to meet the congestion warning speed, the congestion time can be determined; and the pre-set weights for the first moving state, the second moving state, and traffic flow information are first determined based on the second weight index. Using the number of identical states and the congestion time, the weights of each state are adjusted to determine the dynamic weights of the first moving state, the second moving state, and the traffic flow information.

[0101] For example, if the first state is a semi-congested state, the second state is a moving state, and the third state is a moving state; the number of vehicles in the moving state is 2, and the number in the semi-congested state is 1; the weight of the first moving state is 'a', and the current time is 't1', corresponding to the last frame in the continuous frame information; the weight of the second moving state is 'b', and the current time is 't2', corresponding to the eighth frame in the continuous frame information; the weight of the traffic flow information is 'c', and the current time is 't1', corresponding to the last frame in the continuous frame information; and the time to meet the congestion warning speed is 't3'. Here, 't3' is earlier than 't2', and 't2' is earlier than 't1'. In this case, the dynamic weights of the first moving state can be calculated as 'a*(t3-t1)*1', the second moving state as 'b*(t3-t2)*2', and the third moving state as 'c*(t3-t1)*2'. Note that the second moving state is determined by the indicator lights of the vehicle in front, and therefore may only appear in a few frames of the continuous frame information. In practice, for example, the second state corresponding to the second motion state can also be determined by comprehensively judging the second state based on the indicator lights of multiple vehicles ahead determined from consecutive frames, or by judging the second state based on the indicator lights in the frame closest to the current real time.

[0102] Furthermore, based on the dynamic weights of the first motion state, the second motion state, and the traffic flow information, the motion state of the preceding vehicle is determined from the first, second, and third states. Specifically, the state corresponding to the dynamic weight with the highest value is selected as the motion state of the preceding vehicle. For example, if the traffic flow information has the highest dynamic weight among these three dynamic weights, then the third state can be determined as the motion state of the preceding vehicle.

[0103] Furthermore, the preceding vehicle's sound status is determined based on the preceding vehicle's sound information. This involves identifying the obtained sound events, the duration of the event, the frequency and amplitude changes of the event throughout the acquisition process, and their corresponding first weighting indicators as the preceding vehicle's sound information. Then, based on each sound event and its corresponding first weighting indicator, the weight value corresponding to each sound is determined. Finally, based on the magnitude of the weight value and the current traffic jam status indicated by each preceding vehicle's sound information, the preceding vehicle's movement status is determined.

[0104] For example, let's take the preceding vehicle's sound information, including the number of sound events and a first weight index, as an example. The same principle applies to other aspects and can also be used in actual weight calculations, which will not be elaborated further here. The preceding vehicle's sound information includes first information, second information, and third information. The first information is the engine start sound, with a corresponding first weight index of 'd', and the sound event occurs 6 times; the second information is the sound of airflow enhancement during vehicle acceleration, with a corresponding first weight index of 'e', ​​and the sound event occurs 3 times; the third information is the vehicle warning sound, with a corresponding first weight index of 'f', and the sound event occurs 10 times; where 'd' is greater than 'e', ​​which is greater than 'f'. At this point, the total weight sum corresponding to the preceding vehicle's sound information can be obtained as: 6d + 3e + 10f. If the total weight sum corresponding to the preceding vehicle's sound information exceeds the preset total weight, then the preceding vehicle's sound status can be determined as a congestion-end state.

[0105] In this embodiment, the corresponding traffic congestion state is determined based on the first motion state, the second motion state, and traffic flow information, respectively. It is then determined whether these three traffic congestion states are all the same. If they are all the same, the traffic congestion state can be determined as the motion state of the vehicle in front, thus achieving a fast and accurate determination of the motion state of the vehicle in front. Alternatively, if at least one state is different from the other states, dynamic weights corresponding to the first motion state, the second motion state, and traffic flow information are determined based on different factors, and the final motion state of the vehicle in front is determined based on the dynamic weights. This achieves diversified determination of the motion state of the vehicle in front, and also improves the adaptive ability to judge the motion state of the vehicle in front in different scenarios based on the dynamic weights determined by different factors, thereby improving the decision-making quality in different scenarios and reducing the impact of human subjective factors on the results.

[0106] S209. Determine and display the traffic jam status based on the movement and sound of the vehicle in front.

[0107] Specifically, the motion and sound states of the vehicle in front can be directly fused using multimodal methods to determine the traffic congestion status. For example, decision-level fusion or weighted voting can be used to determine whether the vehicle in front is in a traffic congestion state based on its motion and sound states.

[0108] Optionally, the first motion state, second motion state, traffic flow information, and various sound events obtained above can be directly used to determine the current traffic congestion status through decision-level fusion.

[0109] Furthermore, once a traffic jam is identified, the traffic jam status can be displayed on the vehicle's visual screen.

[0110] In this embodiment, the traffic jam status is determined based on the movement and sound status of the vehicle in front, which enables a comprehensive judgment of the traffic jam status from both visual and auditory perspectives. This reduces the problem of judgment errors caused by relying solely on visual or auditory judgments, and achieves a comprehensive and accurate judgment of the traffic jam status.

[0111] Optionally, after identifying and displaying the traffic congestion status, the following may also be included:

[0112] (i) If the traffic jam status is "traffic jam ended", a traffic jam end reminder signal is sent to the central control screen so that the central control screen can announce the traffic jam end status.

[0113] Specifically, if the traffic jam status indicates that the traffic jam has ended, a traffic jam end reminder signal is sent to the central control screen, which then directly announces the end of the traffic jam to the driver. This allows for quick notification of the end of the traffic jam and prompts the driver to start driving as soon as possible, effectively improving driving response speed and traffic efficiency. Optionally, when the traffic jam status is confirmed to be ended, the camera and audio equipment functions activated for the purpose of confirming the traffic jam can be turned off to reduce vehicle energy consumption.

[0114] (ii) If the traffic jam status is traffic jam, then return to the steps of obtaining the continuous frame information captured by the vehicle's camera equipment and the environmental audio signal collected by the vehicle's audio equipment.

[0115] Specifically, if the traffic jam is still ongoing, the process can return to step S201 to obtain continuous frame information and environmental audio signals, thus enabling continuous monitoring until the traffic jam ends.

[0116] In this embodiment, if the traffic jam status indicates that the traffic jam has ended, the central control screen proactively displays the end of the traffic jam status to the user. This prevents drivers from missing the end of the traffic jam due to distraction during long traffic jams, thus avoiding problems such as slow starts and improving the driver's experience. Alternatively, if the traffic jam status indicates that the traffic jam is still in progress, the camera and audio equipment continue to collect relevant information to provide the driver with continuous monitoring of the traffic jam status.

[0117] Figure 3 This is a schematic diagram of a traffic jam status determination device provided in an embodiment of the present invention. Figure 3 As shown, the device is applied to a vehicle and includes:

[0118] The acquisition module 301 is used to acquire continuous frame information captured by the vehicle's camera equipment and environmental audio signals collected by the vehicle's audio equipment when the vehicle's driving speed is detected to meet the congestion warning speed.

[0119] The processing module 302 is used to determine the first motion state and the second motion state of the vehicle in front based on the continuous frame information, and to determine the sound information of the vehicle in front based on the ambient audio signal, wherein the vehicle in front includes at least one vehicle in front of the vehicle.

[0120] The determining module 303 is used to determine the motion state of the preceding vehicle based on the first motion state and the second motion state, and to determine the sound state of the preceding vehicle based on the sound information of the preceding vehicle.

[0121] The display module 304 is used to determine and display the traffic jam status based on the movement and sound status of the vehicle in front.

[0122] Optionally, module 303 is specifically used for:

[0123] The system preprocesses the continuous frame information and the obtained environmental audio signal; it performs target recognition on the preprocessed continuous frame information to identify at least one vehicle in front, and extracts features from the preprocessed environmental audio signal to obtain the vehicle's state sound; for each vehicle in front, it determines a first motion state based on the pixel information of the vehicle in the continuous frame information; and it determines a second motion state based on the vehicle's indicator lights; it inputs the vehicle's state sound into a pre-trained vehicle sound event classifier to determine the vehicle sound event, and determines the vehicle sound information based on the temporal features of the vehicle sound event.

[0124] Optionally, the pixel information includes pixel range and pixel position; based on the pixel information of the preceding vehicle in consecutive frame information, the first motion state is determined, and the determining module 303 is specifically used for:

[0125] For each frame of information in a continuous frame, determine the pixel range and pixel position of the preceding vehicle in that frame; based on the pixel range and pixel position of the preceding vehicle in each frame, determine the first motion state.

[0126] Optionally, the time-domain features include sound duration, frequency variation, and amplitude variation; based on the time-domain features of the preceding vehicle's sound event, the preceding vehicle's sound information is determined, and the determining module 303 is specifically used for:

[0127] Obtain the first weight index corresponding to the preceding vehicle sound event, wherein the first weight index is the weight value corresponding to different time-domain characteristics of different preceding vehicle sound events; determine the preceding vehicle sound information based on the sound duration event, frequency change, amplitude change and the first weight index corresponding to the preceding vehicle sound event.

[0128] Optionally, after acquiring the continuous frame information captured by the vehicle's camera equipment, the processing module 302 is further configured to:

[0129] Based on the degree of pixel intensity change between adjacent frames in continuous frame information, the instantaneous velocity vector of each pixel is calculated; based on the vector direction and vector components of the instantaneous velocity vector, traffic flow information is determined.

[0130] Module 303 is specifically used for:

[0131] The motion state of the vehicle ahead is determined based on the first motion state, the second motion state, and traffic flow information.

[0132] Optionally, module 303 is specifically used for:

[0133] Vehicle motion states are determined for the first motion state, the second motion state, and traffic flow information, respectively, to obtain the first state corresponding to the first motion state, the second state corresponding to the second motion state, and the third state corresponding to the traffic flow information. It is then determined whether the first, second, and third states are all identical. If all three states are identical, the identical state is directly identified as the preceding vehicle's motion state. If at least one of the first, second, and third states is different, the dynamic weights of the first motion state, the second motion state, and the traffic flow information are determined based on the number of identical states, the current time, the time required to meet the congestion warning speed, and a second weighting index. The second weighting index includes pre-set base weights for the first, second, and traffic flow information. Based on the dynamic weights of the first, second, and traffic flow information, the preceding vehicle's motion state is determined from the first, second, and third states.

[0134] Optionally, after determining and displaying the traffic jam status based on the movement and sound of the vehicle in front, the display module 304 is also used for:

[0135] If the traffic jam status is "Traffic jam ended", a traffic jam end reminder signal is sent to the central control screen so that the central control screen can announce the traffic jam end status; if the traffic jam status is "Traffic jam", the process returns to the steps of obtaining continuous frame information captured by the vehicle's camera equipment and the environmental audio signal collected by the vehicle's audio equipment.

[0136] The traffic congestion determination device provided in this embodiment of the invention can execute the traffic congestion determination method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method.

[0137] Figure 4 This is a schematic diagram of the structure of an electronic device in a vehicle provided for an embodiment of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0138] like Figure 4 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0139] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0140] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as the traffic jam state method.

[0141] In some embodiments, the traffic congestion method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the traffic congestion method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the traffic congestion method by any other suitable means (e.g., by means of firmware).

[0142] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0143] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0144] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0145] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0146] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0147] A computing system can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0148] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the traffic jam state determination method provided in any embodiment of this invention.

[0149] In implementing the computer program product, computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0150] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0151] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A traffic jam state determination method characterized by comprising: Applied to vehicles; the method includes: When the vehicle's speed is detected to meet the congestion warning speed, the system acquires continuous frame information captured by the vehicle's camera and environmental audio signals collected by the vehicle's audio equipment. The method involves determining the first and second motion states of the preceding vehicle based on the continuous frame information, and determining the preceding vehicle's sound information based on the ambient audio signal. This includes: preprocessing the continuous frame information and the ambient audio signal; performing target recognition on the preprocessed continuous frame information to identify at least one preceding vehicle, and extracting features from the preprocessed ambient audio signal to obtain the preceding vehicle's state sound; for each preceding vehicle, determining the first motion state based on the preceding vehicle's pixel information in the continuous frame information; and determining the second motion state based on the preceding vehicle's indicator lights; inputting the preceding vehicle's state sound into a pre-trained preceding vehicle sound event classifier to determine preceding vehicle sound events, and determining the preceding vehicle's sound information based on the temporal features of the preceding vehicle sound events; wherein the preceding vehicle includes at least one vehicle in front of the vehicle. The preceding vehicle's motion state is determined based on the first motion state and the second motion state, and the preceding vehicle's sound state is determined based on the preceding vehicle's sound information. The traffic jam status is determined and displayed based on the movement and sound status of the vehicle in front; After acquiring the continuous frame information captured by the vehicle's camera device, the method further includes: calculating the instantaneous velocity vector of a pixel based on the degree of pixel intensity change between every two adjacent frames in the continuous frame information; determining traffic flow information based on the vector direction and vector components of the instantaneous velocity vector; and determining the preceding vehicle's motion state based on the first motion state and the second motion state includes: determining the preceding vehicle's motion state based on the first motion state, the second motion state, and the traffic flow information.

2. The method for determining traffic congestion status according to claim 1, characterized in that, The pixel information includes pixel range and pixel position; Determining the first motion state based on the pixel information of the preceding vehicle in the consecutive frame information includes: For each frame of the continuous frame information, determine the pixel range and pixel position of the preceding vehicle in that frame information; The first motion state is determined based on the pixel range and pixel position of the preceding vehicle in each frame of information.

3. The method for determining traffic congestion status according to claim 1, characterized in that, The time-domain features include sound duration, frequency variation, and amplitude variation; determining the preceding vehicle sound information based on the time-domain features of the preceding vehicle sound event includes: Obtain the first weight index corresponding to the preceding vehicle sound event, wherein the first weight index is the weight value corresponding to different time-domain features of different preceding vehicle sound events; The preceding vehicle sound information is determined based on the sound duration, the frequency change, the amplitude change, and the first weight index corresponding to the preceding vehicle sound event.

4. The method for determining traffic congestion status according to claim 1, characterized in that, Determining the motion state of the vehicle ahead based on the first motion state, the second motion state, and the traffic flow information includes: The vehicle motion state is determined by the first motion state, the second motion state, and the traffic flow information respectively, to obtain the first state corresponding to the first motion state, the second state corresponding to the second motion state, and the third state corresponding to the traffic flow information. Determine whether the first state, the second state, and the third state are all the same; If the first state, the second state, and the third state are all the same, then the consistent state is directly determined as the forward vehicle's motion state. If at least one of the first state, the second state, and the third state is different, then the dynamic weight of the first motion state, the dynamic weight of the second motion state, and the dynamic weight of the traffic flow information are determined based on the number of identical states, the current time, the time when the congestion warning speed is met, and the second weight index. The second weight index includes the pre-set basic weights of the first motion state, the second motion state, and the traffic flow information. Based on the dynamic weights of the first motion state, the second motion state, and the traffic flow information, the motion state of the preceding vehicle is determined from the first state, the second state, and the third state.

5. The method for determining traffic congestion status according to claim 1, characterized in that, After determining and displaying the traffic jam status based on the movement and sound status of the vehicle in front, the process also includes: If the traffic jam status is "traffic jam ended", a traffic jam end reminder signal is sent to the central control screen so that the central control screen announces the end of the traffic jam. If the traffic jam status is a traffic jam, then return to the step of obtaining continuous frame information captured by the vehicle's camera device and the environmental audio signal collected by the vehicle's audio device.

6. A traffic jam status determination device, characterized in that, Applied to vehicles; the device includes: The acquisition module is used to acquire continuous frame information captured by the vehicle's camera equipment and environmental audio signals collected by the vehicle's audio equipment when the vehicle's driving speed is detected to meet the congestion warning speed. A processing module is configured to determine a first motion state and a second motion state of a preceding vehicle based on the continuous frame information, and to determine the preceding vehicle sound information based on the ambient audio signal. The module includes: preprocessing the continuous frame information and the ambient audio signal; performing target recognition on the preprocessed continuous frame information to identify at least one preceding vehicle, and extracting features from the preprocessed ambient audio signal to obtain the preceding vehicle state sound; for each preceding vehicle, determining the first motion state based on the pixel information of the preceding vehicle in the continuous frame information; and determining the second motion state based on the preceding vehicle's indicator lights; inputting the preceding vehicle state sound into a pre-trained preceding vehicle sound event classifier to determine preceding vehicle sound events, and determining the preceding vehicle sound information based on the temporal features of the preceding vehicle sound events; wherein the preceding vehicle includes at least one vehicle in front of the vehicle. The determining module is used to determine the motion state of the vehicle in front of the vehicle based on the first motion state and the second motion state, and to determine the sound state of the vehicle in front of the vehicle based on the sound information of the vehicle in front. The display module is used to determine and display the traffic jam status based on the movement status and sound status of the vehicle in front; After acquiring the continuous frame information captured by the vehicle's camera device, the acquisition module is further configured to: calculate the instantaneous velocity vector of the pixel based on the degree of pixel intensity change between every two adjacent frames in the continuous frame information; determine traffic flow information based on the vector direction and vector components of the instantaneous velocity vector; the step of determining the preceding vehicle's motion state based on the first motion state and the second motion state includes: determining the preceding vehicle's motion state based on the first motion state, the second motion state, and the traffic flow information.

7. A vehicle, characterized in that, include: Electronic devices; The electronic device includes one or more processors; A memory for storing one or more programs that, when executed by one or more processors, cause the one or more processors to implement the traffic congestion state determination method as described in any one of claims 1 to 5.

8. A readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the traffic jam status determination method as described in any one of claims 1 to 5.

Images