Emergency call method, electronic equipment and storage medium

Abstract

The invention provides an emergency call method, electronic equipment and a storage medium. The method comprises the following steps: acquiring an original image of a driver; determining the hand posture of the driver according to the single frame or continuous multiple frames of original images, and / or determining the lip language information of the driver according to the continuous multiple frames of original images; determining the type of the emergency call according to the hand posture of the driver and / or the lip language information; and triggering the emergency call of the category. Through the technical scheme of the invention, the triggering mode of the emergency call is enriched, a special hardware device does not need to be additionally configured, and the triggering efficiency, the triggering precision and the triggering stability of the emergency call are improved.

Description

Technical Field

[0001] The present application relates to the technical field of vehicle emergency calls, and in particular to an emergency call method, electronic equipment, and storage medium. Background Art

[0002] When a vehicle has an accident, such as a collision, the vehicle will automatically trigger an emergency call, or the driver and passengers can actively trigger an emergency call to seek help from the outside world in a timely manner.

[0003] Among them, the method of automatically triggering an emergency call relies on a special detection device, which detects vehicle collision accidents and automatically triggers an emergency call when a vehicle collision accident is detected. However, in some scenarios, the special detection device may be damaged due to a collision accident, causing its detection function to fail, and thus failing to trigger an emergency call in time. The way for drivers and passengers to actively trigger emergency calls is currently mostly through specific physical buttons. When the driver and passengers press the specific physical button, the emergency call is triggered. However, in some scenarios, the driver and passengers are restricted in their movements and may not be able to touch the specific physical button, resulting in the failure to trigger the emergency call in time.

[0004] In view of this, this application is hereby filed. Summary of the Invention

[0005] This application aims to provide an emergency call method, electronic device and storage medium, and proposes a new emergency call method, which enriches the triggering methods of emergency calls, does not require additional configuration of special hardware devices, and improves the triggering efficiency, triggering accuracy and triggering stability of emergency calls.

[0006] In a first aspect, an embodiment of the present application provides an emergency call method, comprising: acquiring an original image of a driver; Determining the driver's hand gesture based on a single frame or multiple consecutive frames of the original image, and / or determining the driver's lip reading information based on multiple consecutive frames of the original image; determining a category of the emergency call based on the driver's hand gesture and / or the lip reading information; Trigger an emergency call of the stated category.

[0007] According to the technical solution provided in the embodiment of the present application, optionally, determining the driver's hand posture based on a single frame or multiple consecutive frames of the original image includes: Performing hand gesture recognition on the single frame of the original image to obtain a corresponding hand gesture, and determining the corresponding hand gesture as the driver's hand gesture; Alternatively, hand gesture recognition is performed on multiple consecutive frames of the original image to obtain the hand gesture corresponding to each frame of the original image. If the number of frames of the original image with the same corresponding hand gesture reaches a threshold, the same hand gesture is determined to be the driver's hand gesture.

[0008] According to the technical solution provided in the embodiment of the present application, optionally, the identifying the detection frame information of the hand area from the original image includes: Cropping the original image into a first intermediate image of a first resolution; Inputting the first intermediate image into a trained hand region detection model to obtain a preset number of reference detection frame information, wherein each reference detection frame information corresponds to a confidence score; Determine the reference detection box information with a confidence score greater than the score threshold as the candidate detection box information; Determine the mapping detection frame information corresponding to the candidate detection frame information in the original image; Determining the mapping detection frame information corresponding to the maximum detection frame from the plurality of mapping detection frame information by a non-maximum suppression algorithm; The mapped detection frame information corresponding to the maximum detection frame is determined as the detection frame information of the hand area.

[0009] According to the technical solution provided in the embodiment of the present application, optionally, determining the corresponding hand posture according to the hand region sub-image includes: Cropping the hand region sub-image into a second intermediate image of a second resolution; The second intermediate image is input into a trained gesture recognition model to obtain a hand posture corresponding to the hand region sub-image.

[0010] According to the technical solution provided in the embodiment of the present application, optionally, determining the driver's lip reading information based on multiple consecutive frames of the original image includes: Performing lip region recognition on a plurality of consecutive frames of the original image to obtain a lip region sub-image corresponding to each frame of the original image; Inputting a lip region sub-image sequence consisting of lip region sub-images corresponding to a plurality of consecutive frames of the original image into a trained lip reading recognition model to obtain character probability distribution information corresponding to each frame of the original image; The character probability distribution information corresponding to each frame of the original image is input into the connection time series classifier CTC to obtain the driver's lip reading information.

[0011] According to the technical solution provided in the embodiment of the present application, optionally, determining the category of the emergency call based on the driver's hand gesture and / or the lip reading information includes: determining a call category associated with the driver's hand gesture as the category of the emergency call; Alternatively, determining the call category indicated by the lip reading information as the category of the emergency call; Alternatively, a union of the call category associated with the driver's hand gesture and the call category indicated by the lip reading information is determined as the category of the emergency call.

[0012] According to the technical solution provided in the embodiment of the present application, optionally, the category of the emergency call includes a call alarm category, a call ambulance category, or a call insurance category; after triggering the emergency call of the category, the emergency call method further includes: Feedback notification that an emergency call has been triggered.

[0013] In a second aspect, an embodiment of the present application further provides an emergency call device, comprising: An acquisition module, used for acquiring an original image of the driver; a first determining module, configured to determine the driver's hand gesture based on a single frame or multiple consecutive frames of the original image, and / or determine the driver's lip reading information based on multiple consecutive frames of the original image; a second determining module, configured to determine a category of the emergency call based on the driver's hand gesture and / or the lip reading information; A calling module is used to trigger an emergency call of the category.

[0014] In a third aspect, an embodiment of the present application further provides an electronic device, comprising: processor and memory; The processor is configured to execute the steps of the emergency call method as described in any embodiment by calling the program or instruction stored in the memory.

[0015] In a third aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a program or instruction, and the program or instruction enables a computer to execute the steps of the emergency call method as described in any embodiment.

[0016] In summary, this application proposes an emergency call method that identifies the driver's hand gestures and / or lip readings based on the driver's original image, then determines the driver's call request based on the driver's hand gestures and / or lip readings, and ultimately triggers a corresponding emergency call based on the driver's call request. Because the driver actively initiates the call request, the accuracy of the emergency call is greatly improved. Redundant emergency calls based on the driver's hand gestures and lip readings improve the stability of the emergency call and do not require the configuration of additional complex hardware devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a process of an emergency call method provided by the embodiment of the present application Figure 1 ; Figure 2 This is a schematic diagram of the structure of a recognition model provided in an embodiment of the present application; Figure 3 This is a process of an emergency call method provided by the embodiment of the present application Figure 2 ; Figure 4 This is a schematic diagram of a functional module provided in an embodiment of the present application; Figure 5 This is a schematic structural diagram of an emergency call device provided in an embodiment of the present application; Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0018] The present application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the relevant invention and are not intended to limit the invention. It should also be noted that, for ease of description, only portions relevant to the invention are shown in the accompanying drawings.

[0019] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0020] Figure 1 This is a flow chart of an emergency call method provided by an embodiment of the present application. This emergency call method is applicable to scenarios where an emergency call is triggered when a vehicle encounters an emergency (e.g., a vehicle collision or a vehicle failure). This emergency call method can be executed by an emergency call device, which can be implemented in the form of software and / or hardware. Figure 1 , the emergency call method specifically includes the following steps: S110: Acquire an original image of the driver.

[0021] Generally, the applicable scenario of the embodiments of the present application is that the driver encounters an emergency while driving a vehicle, such as a vehicle collision accident, or a vehicle malfunction, which restricts the driver's movement. For example, the driver cannot get out of the vehicle to ask for help or cannot reach the physical button for asking for help. At this time, the emergency call method provided by the embodiments of the present application can help the driver to get emergency help.

[0022] Specifically, the original image of the driver is obtained at a certain frequency. When the driver is sitting in the driver's seat and driving the vehicle, the original image is required to include an image of the driver's upper body. The original image can be obtained by a vehicle-mounted camera. For example, the vehicle-mounted camera can be installed at a position such as the A-pillar, which can fully capture the driver without affecting the driver's driving vision.

[0023] S120: Determine the driver's hand gesture based on a single frame or multiple consecutive frames of the original image, and / or determine the driver's lip reading information based on multiple consecutive frames of the original image.

[0024] Specifically, hand gestures may include "scissor hand," "OK," "right palm facing the driver," "left palm facing the driver," "right hand back facing the driver," etc. Different hand gestures represent different call requests. For example, hand gesture 1 represents calling 110, hand gesture 2 represents calling 120, and hand gesture 3 represents calling the insurance company.

[0025] The essence of lip reading information is the text content spoken by the driver. The text content spoken by the driver can also be determined through voice recognition. However, in some noisy environments, the accuracy of voice recognition is not high. Therefore, in the embodiment of the present application, lip reading information is used to identify the driver's call needs.

[0026] S130: Determine the category of the emergency call according to the driver's hand gesture and / or the lip reading information.

[0027] As mentioned above, different hand gestures represent different call needs, for example, hand gesture 1 represents call 110, hand gesture 2 represents call 120, and hand gesture 3 represents calling an insurance company. One call need corresponds to a specific call category. In general, the categories of emergency calls include call alarm category, call ambulance category, or call insurance category. For example, when the driver's hand gesture is identified as hand gesture 1, the category of the emergency call is the alarm category; when the driver's hand gesture is identified as hand gesture 2, the category of the emergency call is the ambulance category; when the driver's hand gesture is identified as hand gesture 3, the category of the emergency call is the insurance category.

[0028] The determining the category of the emergency call based on the driver's hand gesture and / or the lip reading information includes: determining a call category associated with the driver's hand gesture as the category of the emergency call; Alternatively, determining the call category indicated by the lip reading information as the category of the emergency call; Alternatively, a union of the call category associated with the driver's hand gesture and the call category indicated by the lip reading information is determined as the category of the emergency call.

[0029] For example, if the hand gesture recognition result is 0, it indicates dialing 120 and 110; if the hand gesture recognition result is 1, it indicates dialing the insurance company; and if the hand gesture recognition result is 2, it indicates dialing the emergency contact. If the current hand gesture recognition result is 0 and the recognized lip reading information indicates dialing the emergency contact, the final result is determined to be dialing 120, 110, and the emergency contact simultaneously. By identifying the driver's emergency needs from multiple dimensions, the stability and efficiency of emergency calls are improved, allowing drivers to receive effective assistance in a timely manner.

[0030] S140: Trigger an emergency call of the category.

[0031] The emergency call method provided in the embodiments of the present application identifies the driver's hand gestures and / or lip readings based on the driver's original image, and then determines the driver's call request based on the driver's hand gestures and / or lip readings, ultimately triggering the corresponding emergency call based on the driver's call request. Because the driver actively initiates the call request, the accuracy of the emergency call is greatly improved. Redundant emergency calls based on the driver's hand gestures and lip readings improve the stability of the emergency call, and do not require the configuration of additional complex hardware devices.

[0032] In some embodiments, determining the driver's hand posture based on a single frame or multiple consecutive frames of the original image includes: Hand gesture recognition is performed on the single frame of the original image to obtain a corresponding hand gesture, and the corresponding hand gesture is determined as the hand gesture of the driver.

[0033] Alternatively, hand gesture recognition is performed on multiple consecutive frames of the original image to obtain the hand gesture corresponding to each frame of the original image. If the number of frames of the original image with the same corresponding hand gesture reaches a threshold, the identical hand gesture is determined as the driver's hand gesture. Compared to determining the driver's hand gesture based on a single frame of the original image, determining the driver's hand gesture based on multiple consecutive frames of the original image allows for more accurate hand gesture determination and avoids misidentification of the preset hand gesture due to unconscious hand movements of the driver. For example, if the driver accidentally makes a fist gesture in front of his chest, which just indicates that the driver needs to make an emergency call to 110, a false trigger will occur. However, by performing hand gesture recognition on multiple consecutive frames (for example, 20 frames) of original images, if the driver's fist gesture is recognized in 15 frames of the original images (that is, if the number of frames of the original images with the same corresponding hand gesture reaches a threshold, the same hand gesture is determined to be the driver's hand gesture), it is determined that the preset hand gesture is recognized. This hand gesture is not made unconsciously by the driver, but is intentional by the driver, and there is indeed a need to make an emergency call to 110, thereby achieving the purpose of improving the accuracy and reliability of emergency calls.

[0034] In some embodiments, performing hand gesture recognition on a single frame of the original image to obtain a corresponding hand gesture includes: identifying detection frame information of the hand area from the original image; extracting a hand area sub-image from the original image based on the detection frame information; and determining the corresponding hand gesture based on the hand area sub-image.

[0035] The step of identifying the detection frame information of the hand area from the original image includes the following sub-steps: 121. Crop the original image into a first intermediate image with a first resolution.

[0036] Typically, the resolution of the original image is 1920*1080. To reduce computing resource consumption and improve detection efficiency, the original image is cropped before detection to obtain a first intermediate image of a first resolution (e.g., 192*320). The number of pixels in the first intermediate image is less than that in the original image.

[0037] Optionally, the original image is cropped based on the ARM NEON technology of the 8155 platform. Compared with image cropping using the OpenCV library, higher cropping efficiency can be achieved through the ARM NEON technology, thereby improving the real-time performance and efficiency of this emergency call.

[0038] 122. Input the first intermediate image into a trained hand region detection model to obtain a preset number of reference detection frame information, wherein each reference detection frame information corresponds to a confidence score.

[0039] 123. Determine the reference detection box information whose confidence score is greater than the score threshold as the candidate detection box information.

[0040] 124. Determine mapping detection frame information corresponding to the candidate detection frame information in the original image.

[0041] Specifically, the candidate detection frame information includes the width and height of the detection frame, as well as the vertex coordinates of the upper left corner of the detection frame. This is information relative to the first intermediate image (192*320). Furthermore, the candidate detection frame information needs to be converted to information relative to the original image (1920*1080), that is, the detection frame information is mapped to obtain the vertex coordinates of the upper left corner of the detection frame and the width and height in the original image.

[0042] 125. Determine mapping detection box information corresponding to a maximum detection box from the plurality of mapping detection box information by using a non-maximum suppression algorithm.

[0043] Specifically, the confidence scores are sorted from high to low, with the goal of prioritizing the detection frame most likely to be a hand. The detection frame represented by the mapped detection frame with the highest confidence score is then selected as the current retained frame. The intersection-of-union (IoU) of the current retained frame and all other detection frames is calculated. If the IoU of a detection frame with the current retained frame exceeds a set threshold (e.g., 0.5), it is considered to have excessive overlap with the current retained frame and is removed. The remaining mapped detection frame is then determined to be the mapped detection frame corresponding to the largest detection frame.

[0044] By using a non-maximum suppression algorithm to determine the mapped detection frame information corresponding to the maximum detection frame from multiple mapped detection frame information, redundant detection frames can be eliminated. Due to factors such as hand posture, angle, and occlusion, the model may generate multiple overlapping detection frames in the same hand area. These redundant detection frames not only increase the computational complexity of subsequent processing but can also lead to incorrect judgments of hand position and posture. By effectively eliminating these redundant detection frames, the detection results can be more accurate.

[0045] 126. Determine the mapped detection frame information corresponding to the maximum detection frame as the detection frame information of the hand region.

[0046] Furthermore, determining the corresponding hand posture according to the hand region sub-image includes: The hand area sub-image is cropped into a second intermediate image of a second resolution; and the second intermediate image is input into a trained gesture recognition model to obtain a hand posture corresponding to the hand area sub-image.

[0047] In some embodiments, determining the driver's lip reading information based on multiple consecutive frames of the original image includes: Performing lip region recognition on a plurality of consecutive frames of the original image to obtain a lip region sub-image corresponding to each frame of the original image; Inputting a lip region sub-image sequence consisting of lip region sub-images corresponding to a plurality of consecutive frames of the original image into a trained lip reading recognition model to obtain character probability distribution information corresponding to each frame of the original image; The character probability distribution information corresponding to each frame of the original image is input into the connection time series classifier CTC to obtain the driver's lip reading information.

[0048] For example, a video sequence consisting of T frames is input. For each frame, the model outputs a vector of length |V|, where |V| is the vocabulary size (including the blank label _ and other possible characters). CTC allows for repeated characters and blank labels to flexibly handle input and output of varying lengths. CTC calculates the probabilities of all possible paths that can be collapsed into the same output sequence, removes repeated characters and blank labels from the path, obtains the final output sequence, and selects the one with the highest probability as the final output. For example, the path "h__e_l_ll_o_" can be collapsed into "hello". Optionally, the lip reading recognition model outputs a result of 0 when it recognizes 120 or 110, a result of 1 when it recognizes "insurance", and a result of 2 when it recognizes "emergency contact".

[0049] Assume that the hand gesture recognition result is 0, which means dialing 120 and 110; the hand gesture recognition result is 1, which means dialing the insurance company; and the hand gesture recognition result is 2, which means dialing the emergency contact.

[0050] If the hand gesture recognition result is 0 and the lip reading recognition model output is 1, the system will dial 120, 110, and the insurance company simultaneously. If the hand gesture recognition result is 0 and the lip reading recognition model output is 2, the system will dial 120, 110, and the emergency contact simultaneously. If the hand gesture recognition result is 1 and the lip reading recognition model output is 2, the system will dial the insurance company and the emergency contact simultaneously.

[0051] Furthermore, once an emergency call is triggered through gestures or lip reading, the app immediately triggers the Chime sound effect module. The system plays a rapid Chime sound effect through the car's speakers, notifying the user that the emergency call has been successfully triggered and providing further action instructions (such as "stay calm" and "wait for help"). Different Chime sound effects can be designed based on different gestures and scenarios (users can also pre-define settings through the interactive interface).

[0052] At the same time, the system records the time of each emergency call, gesture recognition results, and sent location information for subsequent analysis and improvement.

[0053] In summary, the emergency call method provided in the embodiment of the present application includes the following specific steps: Predefined emergency call SOS gestures: Select a simple, memorable gesture as the SOS gesture. This gesture is available through the SOS drop-down menu on the central control screen. For example, "crossing your fists at a 45-degree angle" indicates calling 120 or 110 for emergency assistance; "holding your right fist at a 45-degree angle" indicates calling the insurance hotline; and "holding your left fist at a 45-degree angle" indicates calling an emergency contact. These SOS gestures are examples only and can be customized by the user. However, they should be carefully designed to avoid confusion with other everyday gestures to ensure accurate recognition.

[0054] Image acquisition: A high-definition camera with a resolution of 1920*1080 is installed inside the vehicle, located at the A-pillar, to accurately capture the driver's hand movements. When the vehicle is powered on, the system begins operation, starting a camera acquisition thread 1 to capture images of the driver and place them in a queue. If an image already exists in the queue, the queue is cleared and the latest image is placed, ensuring that the latest frame is always processed.

[0055] Design the network structure and train the gesture recognition model and lip reading recognition model: refer to Figure 2 The schematic diagram of a model structure shown in FIG. 1 can be used to train the model structure to obtain the gesture recognition model and the lip reading recognition model. Figure 2 As shown in the figure, the model includes the following key components: convolutional layer (Conv) and activation function (ReLU), batch normalization (BatchNormalization), reshape layer (Reshape), and normalization layer (Softmax). Convolutional layer (Conv) and activation function (ReLU) are used for feature extraction and nonlinear transformation.

[0056] First convolutional layer: Figure 2 W<16x1x3x3> indicates that this convolutional layer has one input channel (a grayscale image) and produces 16 output channels (feature maps). It uses a 3x3 convolution kernel to detect local features in the input image, such as edges or textures. Each 3x3 convolution kernel slides over the entire image, calculating the dot product within the local region to generate a new feature map.

[0057] Second convolutional layer: W<16x16x3x3> describes a convolutional layer with 16 input channels and 16 output channels. Building on the first convolutional layer, the second convolutional layer takes as input the 16 feature maps generated by the previous convolutional layer and outputs new features learned based on these feature maps. In this way, the model is able to gradually build more complex and abstract feature representations.

[0058] The activation function (ReLU) introduces nonlinearity, allowing the network to fit more complex functional relationships. For any input value x, all negative values are set to zero, while positive values remain unchanged. This is done to speed up training by setting some neuron outputs to zero, thereby simplifying the model structure.

[0059] About Batch Normalization, Figure 2 mid-scale <16> 、B <16> 、mean <16> var <16> are learnable parameters that allow scaling and translation of the normalized data; mean and var store the mean and variance during the run, respectively, for data normalization in the subsequent inference phase. <16> 、B <16> 、mean <16> var <16> This section describes the parameter configuration of the Batch Normalization layer, indicating that this layer processes data with 16 feature channels. Batch Normalization adjusts the data distribution of each channel so that its mean approaches 0 and its variance approaches 1, thereby ensuring data consistency. This aims to reduce training instabilities caused by variations in the input data distribution and improve the learning efficiency and stability of the model.

[0060] The Reshape layer flattens the multidimensional feature map into a one-dimensional vector to meet the requirements of the fully connected layer. The Reshape operation converts the feature map into a [batch_size, channels * height * width] format for subsequent fully connected layer processing. This operation does not change the content of the data, but only its arrangement, ensuring the integrity and accuracy of the information.

[0061] The normalization layer (Softmax) converts the output of the final layer into a probability distribution, making it ideal for multi-classification tasks. Assuming an 8-dimensional output vector after the fully connected layer (Gemm matrix multiplication), Softmax converts it into predicted probabilities for the eight categories, which sum to 1. For this embodiment of the application, the output of the final layer is in the form of hand posture IDs and corresponding scores. The hand posture represented by the hand posture ID with the highest score is determined as the final recognized hand posture.

[0062] The image resolution captured by the camera is 1920x1080. To meet the requirements of model training, the image needs to be scaled to 128x128 before being input into the model for training. In this process: The first convolution extracts 16 different basic feature maps to help identify basic patterns in the input image. The second convolution layer uses 16 new 3x3 convolution kernels based on the 16 feature maps generated by the previous convolution layer to further learn higher-level feature representations. As the network depth increases, through a series of similar operations (from Figure 2 As can be seen in the figure, the resulting vector contains 640 features. Next, through Gemm matrix multiplication (the core of the fully connected layer), this 640-dimensional feature vector is converted into an 8-dimensional output vector. This involves multiplication with an 8×640×640 weight matrix and the addition of a bias vector of length 8. Finally, the Softmax function is applied to convert this 8-dimensional output vector into a probability distribution between 0 and 1, representing the probability of each category being correctly classified.

[0063] Based on this, the hand gesture recognition process involves feeding the captured image into a pre-trained gesture recognition model, extracting hand features and identifying whether it represents a pre-defined SOS gesture. Trained on a large dataset, the gesture recognition model accurately recognizes a variety of static gestures and is specifically optimized for SOS gestures. Hardware acceleration and algorithm optimization ensure that gesture recognition is completed in milliseconds, meeting real-time requirements. The system can run continuously in the background, but the emergency call module is activated only when a pre-defined gesture is detected, minimizing system resource usage.

[0064] Specifically, the system starts thread 1 for algorithm processing. It first initializes the hand detection threshold nmsHandsInnerThreshold = 0.95 and the number of inference frames output by the algorithm, gridCells = 4620. After thread 1 completes initialization, it retrieves the captured 1920*1080 image from the queue and crops it to a size of 128*128. The ARM NEON technology used in the 8155 platform makes image cropping faster than using the OpenCV library. The 128*128 image is then fed into the hand region detection model interface, where the coordinates and confidence scores of all hand detection frames are inferred. Hand detection frames with confidence scores less than 0.95 are filtered out. The remaining hand detection frames are then converted to the 1920*1080 image coordinates to obtain the (x, y, w, h) coordinates, representing the upper-left corner coordinates, width, and height of the hand detection frame. The 1920*1080 image retrieved from the queue is cropped to 112*112 resolution based on the hand detection bounding box. The ARM NEON technology used in the 8155 platform makes image cropping faster than using the OpenCV library. The 112*112 image is fed into the model for inference, and the inferred probabilities of various gestures are output as IDs and scores. The ID value ranges from 0 to 8, where 0 represents other gestures, 1 represents thumbs-left, 2 represents thumbs-right, 3 represents thumbs-up, 4 represents "shush," 5 represents palm facing forward, 6 represents left hand clasped at a 45-degree angle, 7 represents right hand clasped at a 45-degree angle, and 8 represents arms crossed at a 45-degree angle. The gesture with the highest score is the final gesture.

[0065] If the SOS gesture recognition model outputs a 2, it indicates a call to the emergency contact; if the SOS gesture recognition model outputs a 1, it indicates a call to the insurance number; if the SOS gesture recognition model outputs a 0, it indicates a call to both 120 and 110. You can also choose whether to add multi-frame smoothing based on actual conditions. This means setting an expiration time, such as 1500ms, to ensure that the current result is valid only within 1500ms. Then, set a multi-frame range, such as 20 frames. If the SOS gesture is detected in 10 of 20 consecutive frames, it is considered an SOS gesture; otherwise, it is considered a non-SOS gesture to prevent false detections.

[0066] If the SOS gesture is detected, continue with the subsequent operation; if not, repeat the gesture recognition operation.

[0067] Lip reading recognition, like gesture recognition, outputs a probability distribution of characters corresponding to each frame. Post-processing uses a Connectionist Temporal Classification (CTC) decoder to convert the resulting image into specific words or sentences. For example, given a video sequence consisting of T frames, the lip reading model outputs a vector of length |V| for each frame, where |V| is the vocabulary size (including the blank label _ and other possible characters). CTC allows for repeated characters and blank labels, allowing for flexible handling of input and output lengths. The probabilities of all possible paths must be calculated, so that these paths can be collapsed into the same output sequence. Repeated characters and blank labels are removed from the paths to obtain the final output sequence, and the one with the highest probability is selected as the final output. For example, the path "h__e_l_ll_o_" can be collapsed into "hello."

[0068] When the lip reading recognition model recognizes 120 or 110, it outputs the result 0; when it recognizes "insurance", it outputs the result 1; when it recognizes "emergency contact", it outputs the result 2.

[0069] Take the union of the inference results of the gesture recognition model and the lip reading recognition model. If the output is {0}, it means calling 120 and 110. If the output is {1}, it means the user needs to call the insurance company. If the output is {2}, it means calling the emergency contact.

[0070] If the output result is {0, 1}, it means calling 120, 110 and the insurance company.

[0071] If the output result is {0, 2}, it means dialing 120, 110 and the emergency contact number.

[0072] If the output result is {1, 2}, it means calling the insurance company and the emergency contact.

[0073] If the output result is {0, 1, 2}, it means calling 120, 110, the insurance company number, and the emergency contact number.

[0074] Audio feedback: Once an SOS result is recognized through gestures and lip reading, the app immediately triggers the sound effects module via the bus. The system plays a rapid sound effect through the car's speakers, notifying the user that the emergency call has been successfully triggered and providing further action instructions (such as "stay calm" and "wait for help"). Different sound effects are designed based on different gestures and scenarios (users can pre-define and select them through the interface).

[0075] For further reference, see Figure 3The flowchart of an emergency call method shown includes the following steps: 310. Initialize the hand region detection model to set the number of outputs to 4620 and the score threshold to 0.95.

[0076] 320. Take out the latest captured original image of the driver from the camera image storage queue.

[0077] 330. Process the original image into a first intermediate image of 192*320.

[0078] 340. Input the first intermediate image into a hand region detection model to obtain an output vector.

[0079] 350. Determine whether the size of the output vector is equal to 4620. If not, end the process. If so, continue to step 360.

[0080] 360. Filter out elements with confidence scores less than 0.95 to obtain candidate detection box information.

[0081] 370. Restore the candidate detection box information to the corresponding mapping detection box information in the original image.

[0082] 380. Determine the mapping detection box information corresponding to the maximum detection box from the multiple mapping detection box information by using a non-maximum suppression algorithm.

[0083] 390. Extract the detection frame information of the hand area from the original image according to the mapped detection frame information corresponding to the maximum detection frame, and crop it into a second intermediate image of 128*128.

[0084] 400. Input the second intermediate image into the recognition model to obtain a first output result.

[0085] 410. Determine the emergency call category based on the lip reading information recognized in the same time period.

[0086] 420. Trigger an emergency call and play a sound effect to notify the driver that the emergency call has been triggered.

[0087] Therefore, the embodiment of the present application involves three modules, such as Figure 4 As shown, they are an emergency call detection module 41, a sound effect module 42 and an alarm module 43.

[0088] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides an emergency call device.

[0089] refer to Figure 5The emergency call device includes: an acquisition module 510, used to acquire the original image of the driver; a first determination module 520, used to determine the driver's hand posture based on a single frame or multiple consecutive frames of the original image, and / or, determine the driver's lip reading information based on multiple consecutive frames of the original image; a second determination module 530, used to determine the category of the emergency call based on the driver's hand posture and / or the lip reading information; a call module 540, used to trigger the emergency call of the category.

[0090] For the convenience of description, the above devices are described as being divided into various modules according to their functions. Of course, when implementing this application, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0091] The apparatus of the above embodiment is used to implement the corresponding emergency call method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0092] For the convenience of description, the above devices are described as being divided into various modules according to their functions. Of course, when implementing this application, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0093] The apparatus of the above embodiment is used to implement the corresponding emergency call method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0094] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 6 As shown, the electronic device 500 includes one or more processors 501 and a memory 502 .

[0095] The processor 501 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 500 to perform desired functions.

[0096] The memory 502 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), a hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 501 may execute the program instructions to implement the emergency call method of any embodiment of the present application described above and / or other desired functions. Various contents such as initial external parameters, thresholds, etc. may also be stored in the computer-readable storage medium.

[0097] In one example, electronic device 500 may further include an input device 503 and an output device 504, which are interconnected via a bus system and / or other connection mechanisms (not shown). Input device 503 may include, for example, a keyboard, a mouse, etc. Output device 504 may output various information to the outside, including warning information, braking force, etc. Output device 504 may include, for example, a display, a speaker, a printer, a communication network, and remote output devices connected thereto.

[0098] Of course, to simplify, Figure 6 Only some of the components related to the present application in the electronic device 500 are shown, and components such as buses, input / output interfaces, etc. are omitted. In addition, the electronic device 500 may further include any other appropriate components according to specific application scenarios.

[0099] In addition to the above methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions. When the computer program instructions are executed by a processor, the processor executes the steps of the emergency call method provided by any embodiment of the present application.

[0100] The computer program product may be written in any combination of one or more programming languages to implement the program code for performing the operations of the embodiments of the present application, including object-oriented programming languages such as Java, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0101] In addition, an embodiment of the present application may also be a computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are executed by a processor, the processor executes the steps of the emergency call method provided by any embodiment of the present application.

[0102] The computer-readable storage medium may be any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may include, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0103] It should be noted that the terms used in this application are only for describing specific embodiments and are not intended to limit the scope of this application. As shown in the specification and claims of this application, unless the context clearly indicates an exception, the words "one", "an", "a kind of" and / or "the" do not specifically refer to the singular and may also include the plural. The terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method or device. In the absence of further restrictions, the elements defined by the sentence "comprise a..." do not exclude the presence of other identical elements in the process, method or device comprising the elements.

[0104] It should also be noted that the terms "center", "up", "down", "left", "right", "vertical", "horizontal", "inside", "outside", etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on this application. Unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", etc. should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or an indirect connection through an intermediate medium, or it can be a communication between the internal parts of two elements. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0105] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. The above is only the preferred implementation method of this application. It should be pointed out that due to the limitations of textual expression, there are objectively infinite specific structures. For ordinary technicians in this technical field, without departing from the principles of the present invention, they can also make several improvements, modifications or changes, and can also combine the above technical features in an appropriate manner; these improvements, modifications, changes or combinations, or the direct application of the inventive concept and technical solution to other occasions without improvement, should be regarded as the scope of protection of this application.

Claims

1. An emergency call method, characterized in that: include: Obtaining the original image of the driver; Determining the driver's hand gesture based on a single frame or multiple consecutive frames of the original image, and / or determining the driver's lip reading information based on multiple consecutive frames of the original image; determining a category of the emergency call based on the driver's hand gesture and / or the lip reading information; Trigger an emergency call of the stated category.

2. The method according to claim 1, characterized in that The determining the driver's hand posture based on a single frame or multiple consecutive frames of the original image includes: Performing hand gesture recognition on the single frame of the original image to obtain a corresponding hand gesture, and determining the corresponding hand gesture as the driver's hand gesture; Alternatively, hand gesture recognition is performed on multiple consecutive frames of the original image to obtain the hand gesture corresponding to each frame of the original image. If the number of frames of the original image with the same corresponding hand gesture reaches a threshold, the same hand gesture is determined to be the driver's hand gesture.

3. The method according to claim 2, characterized in that The performing hand gesture recognition on the single frame of the original image to obtain the corresponding hand gesture includes: Identifying detection frame information of a hand area from the original image; Extracting a hand region sub-image from the original image according to the detection frame information; A corresponding hand posture is determined according to the hand region sub-image.

4. The method according to claim 3, characterized in that The identifying the detection frame information of the hand area from the original image includes: Cropping the original image into a first intermediate image of a first resolution; Inputting the first intermediate image into a trained hand region detection model to obtain a preset number of reference detection frame information, wherein each reference detection frame information corresponds to a confidence score; Determine the reference detection box information with a confidence score greater than the score threshold as the candidate detection box information; Determine the mapping detection frame information corresponding to the candidate detection frame information in the original image; Determining the mapping detection frame information corresponding to the maximum detection frame from the plurality of mapping detection frame information by a non-maximum suppression algorithm; The mapped detection frame information corresponding to the maximum detection frame is determined as the detection frame information of the hand area.

5. The method according to claim 3, characterized in that The determining the corresponding hand posture according to the hand region sub-image includes: Cropping the hand region sub-image into a second intermediate image of a second resolution; The second intermediate image is input into a trained gesture recognition model to obtain a hand posture corresponding to the hand region sub-image.

6. The method according to claim 1, characterized in that The determining of the driver's lip reading information based on the continuous multiple frames of the original image includes: Performing lip region recognition on a plurality of consecutive frames of the original image to obtain a lip region sub-image corresponding to each frame of the original image; Inputting a lip region sub-image sequence consisting of lip region sub-images corresponding to a plurality of consecutive frames of the original image into a trained lip reading recognition model to obtain character probability distribution information corresponding to each frame of the original image; The character probability distribution information corresponding to each frame of the original image is input into the connection time series classifier CTC to obtain the driver's lip reading information.

7. The method according to claim 1, characterized in that The determining the category of the emergency call based on the driver's hand gesture and / or the lip reading information includes: determining a call category associated with the driver's hand gesture as the category of the emergency call; Alternatively, determining the call category indicated by the lip reading information as the category of the emergency call; Alternatively, a union of the call category associated with the driver's hand gesture and the call category indicated by the lip reading information is determined as the category of the emergency call.

8. The method according to claim 1, characterized in that The emergency call category includes an alarm call category, an ambulance call category, an insurance call category, or an emergency contact call category; After triggering the emergency call of the category, the emergency call method further includes: Feedback notification that an emergency call has been triggered.

9. An electronic device, characterized in that: The electronic device comprises: processor and memory; The processor is configured to execute the steps of the emergency call method according to any one of claims 1 to 8 by calling the program or instruction stored in the memory.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a program or instruction, and the program or instruction enables a computer to execute the steps of the emergency call method according to any one of claims 1 to 8.

Images