Signal lamp prompting method, device and equipment based on wearable equipment and storage medium

Abstract

The invention discloses a signal lamp prompting method, device and equipment based on wearable equipment and a storage medium, relates to the technical field of wearable equipment, and discloses the signal lamp prompting method based on the wearable equipment. The wearable device comprises a device supporting piece, a control mainboard and a loudspeaker which are located in the device supporting piece, a device frame body and a camera module located on the device frame body. The method comprises the steps of determining a current prompt mode in response to a prompt instruction of a traffic signal lamp, and performing feature recognition on a current image frame according to an image recognition model corresponding to the mode to obtain a signal lamp recognition result; when the signal lamp identification result is that the traffic signal lamp exists in the current image frame, determining the signal lamp state of the current image frame; and generating a signal lamp prompt signal according to the signal lamp state and broadcasting through a loudspeaker. Through the mode, a driver or a pedestrian can be accurately prompted, the condition of running the red light is avoided, the occurrence rate of traffic accidents is further reduced, and the safety of a user is guaranteed.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of wearable devices, in particular to a signal light prompting method and device based on a wearable device, a device, and a storage medium. BACKGROUND

[0002] In recent years, with the rapid development of the economy and society, the number of private cars has increased rapidly, and road safety problems have become increasingly prominent. In particular, ignoring traffic signals and excessive fatigue of users can cause driving to be in a trance, and then forget to look at the signal light prompt, leading to red light running, and even traffic accidents.

[0003] The above content is only used to assist in understanding the technical solutions of the present application and does not represent the acknowledgement of the above content as prior art. SUMMARY

[0004] The main purpose of the present application is to provide a signal light prompting method, device, equipment and storage medium based on a wearable device, aiming to solve the technical problem that the user cannot be prompted by the traffic signal light in the prior art.

[0005] To achieve the above purpose, the present application provides a signal light prompting method based on a wearable device, which is applied to the control mainboard of the wearable device, the wearable device comprising a device support, a control mainboard, a loudspeaker located inside the device support, a device frame body, and a camera module located on the device frame body, the control mainboard being located inside the device support;

[0006] The method comprises:

[0007] In response to a prompt instruction of a traffic signal light, a current prompt mode is determined;

[0008] According to the image recognition model corresponding to the current prompt mode, the current image frame collected by the camera module is subjected to feature recognition, and a signal light recognition result is obtained;

[0009] When the signal light recognition result is that there is a traffic signal light in the current image frame, the signal light state of the current image frame is determined;

[0010] According to the signal light state, a signal light prompt signal is generated, and the loudspeaker is controlled to broadcast the signal light prompt signal.

[0011] In an embodiment, the step of performing feature recognition on the current image frame collected by the camera module according to the image recognition model corresponding to the current prompt mode to obtain a signal light recognition result comprises:

[0012] When the current prompt mode is pedestrian prompt mode, the first recognition model is called to perform feature recognition on the current image frame captured by the camera module to obtain the traffic light recognition result. The first recognition model is obtained by the terminal training the target detection model with multiple first training images and then lightweighting the trained target detection model. Each first training image is obtained by annotating the traffic light arrangement information and traffic light display form in each sample traffic light image under different environments.

[0013] When the current prompt mode is the driver prompt mode, the second recognition model is invoked to perform feature recognition on the current image frame captured by the camera module to obtain the traffic light recognition result.

[0014] In one embodiment, the step of determining the traffic light status of the current image frame when the traffic light recognition result indicates that a traffic light exists in the current image frame includes:

[0015] When the traffic light recognition result indicates that there is a traffic light in the current image frame, the current image frame is input to the state recognition model. The state recognition model is obtained by the terminal training a convolutional neural network with multiple second training images and then lightweighting the trained convolutional neural network. Each second training image is obtained by labeling the color state of the traffic light in each sample traffic light image under different environments.

[0016] The color state of the current image frame is determined by performing color state recognition on the state recognition model.

[0017] In one embodiment, the step of generating a traffic light prompt signal based on the traffic light status and controlling the loudspeaker to broadcast the traffic light prompt signal includes:

[0018] Acquire the broadcast alert sound effect selected by the user through the terminal;

[0019] Find the prompt sound effect corresponding to the status of the traffic light in the broadcast warning sound effects;

[0020] The system generates a signal light prompt based on the sound effect and controls the speaker to broadcast the signal light prompt.

[0021] In one embodiment, the wearable device further includes an audio sub-board located inside the device support;

[0022] The step of generating a traffic light prompt signal based on the traffic light status and controlling the loudspeaker to broadcast the traffic light prompt signal includes:

[0023] Generate a traffic light prompt signal based on the status of the traffic light;

[0024] controlling the audio sub-board to perform audio processing on the signal lamp prompt signal to obtain a processed signal lamp prompt signal;

[0025] controlling the loudspeaker to broadcast the processed signal lamp prompt signal according to a broadcast frequency.

[0026] In an embodiment, before the step of determining the current prompt mode in response to the prompt instruction of the traffic signal lamp, the method further comprises:

[0027] receiving a state detection image collected by the camera module when receiving the prompt instruction of the traffic signal lamp sent by the user;

[0028] determining a current device visual angle of the wearable device according to the state detection image;

[0029] determining a current device state of the wearable device according to the current device visual angle;

[0030] when the current device state is a normal wearing state, performing the step of determining the current prompt mode in response to the prompt instruction of the traffic signal lamp.

[0031] In an embodiment, before the step of performing feature recognition on a current image frame collected by the camera module according to an image recognition model corresponding to the current prompt mode to obtain a signal lamp recognition result, the method further comprises:

[0032] performing performance detection on the lightweight recognition model when receiving the lightweight recognition model sent by the terminal;

[0033] when the performance detection result is a preset qualified result, deploying the lightweight recognition model according to an operating system supported by the wearable device to obtain an image recognition model.

[0034] In addition, to achieve the above-mentioned purpose, the present application further provides a signal lamp prompt device based on a wearable device, which comprises: a processing module, configured to determine a current prompt mode in response to a prompt instruction of a traffic signal lamp;

[0035] an identification module, configured to perform feature recognition on a current image frame collected by the camera module according to an image recognition model corresponding to the current prompt mode to obtain a signal lamp recognition result;

[0036] the processing module is configured to determine a signal lamp state of the current image frame when the signal lamp recognition result is that there is a traffic signal lamp in the current image frame.

[0037] The control module is configured to generate a signal light prompt signal according to the signal light state, and control the loudspeaker to broadcast the signal light prompt signal.

[0038] In addition, to achieve the above object, the present application also provides a wearable device, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the wearable device-based signal light prompting method.

[0039] In addition, to achieve the above object, the present application also provides a storage medium, which is a computer-readable storage medium, and the storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the wearable device-based signal light prompting method.

[0040] The present application provides a wearable device-based signal light prompting method, which is applied to a control mainboard of the wearable device, and the wearable device comprises a device support, a control mainboard, a loudspeaker located inside the device support, a device frame body and a camera module located on the device frame body, and the control mainboard is located inside the device support; the method comprises the following steps: determining a current prompting mode in response to a prompting instruction of a traffic signal light; performing feature recognition on a current image frame collected by the camera module according to an image recognition model corresponding to the current prompting mode to obtain a signal light recognition result; determining a signal light state of the current image frame when the signal light recognition result is that the current image frame contains a traffic signal light; generating a signal light prompt signal according to the signal light state, and controlling the loudspeaker to broadcast the signal light prompt signal. In this way, the wearable device can accurately recognize the state of the traffic signal light, generate a corresponding prompt signal for broadcasting, prompt the driver or the pedestrian, avoid the situation of running a red light, further reduce the incidence of traffic accidents, and ensure the safety of the user. BRIEF DESCRIPTION OF DRAWINGS

[0041] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and serve to explain the principles of the present application together with the specification.

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, for those skilled in the art, other drawings can also be obtained without creative labor based on these drawings.

[0043] Figure 1A flowchart provided by the signal lamp prompting method based on a wearable device according to Embodiment 1 of the present application;

[0044] Figure 2 A wearable device structure diagram of the signal lamp prompting method based on a wearable device according to Embodiment 1 of the present application;

[0045] Figure 3 A flowchart provided by the signal lamp prompting method based on a wearable device according to Embodiment 2 of the present application;

[0046] Figure 4 A flowchart provided by the signal lamp prompting method based on a wearable device according to Embodiment 2 of the present application;

[0047] Figure 5 A module structure diagram of the signal lamp prompting device based on a wearable device according to Embodiment of the present application;

[0048] Figure 6 A device structure diagram of the hardware running environment involved in the signal lamp prompting method based on a wearable device according to Embodiment of the present application.

[0049] The object implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0050] It should be understood that the specific embodiments described herein are merely intended to explain the technical solutions of the present application, and are not intended to limit the present application.

[0051] In order to better understand the technical solutions of the present application, the specific embodiments will be described in detail below with reference to the drawings and the specific embodiments.

[0052] The main solution of the present application is: in response to the prompting instruction of the traffic signal lamp, determining the current prompting mode; according to the image recognition model corresponding to the current prompting mode, performing feature recognition on the current image frame collected by the camera module to obtain a signal lamp recognition result; when the signal lamp recognition result is that there is a traffic signal lamp in the current image frame, determining the signal lamp state of the current image frame; generating a signal lamp prompting signal according to the signal lamp state, and controlling the loudspeaker to broadcast the signal lamp prompting signal.

[0053] In recent years, with the rapid development of economic society, the number of private cars has increased rapidly, and road safety problems have become more and more prominent. Especially ignoring the signal lamp, and the user's excessive fatigue will cause the driver to be absent-minded, and then forget to look at the signal lamp prompt, leading to the situation of running a red light, and even causing traffic accidents.

[0054] The application can accurately identify the state of a traffic signal lamp through a wearable device, and generate a corresponding prompt signal for broadcasting to prompt a driver or a pedestrian, so as to avoid the situation of running a red light, and further reduce the incidence of traffic accidents, and guarantee the safety of the user.

[0055] It should be noted that the execution subject of the embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone and the like, or an electronic device, a wearable device and the like capable of realizing the above functions. The following takes the wearable device as an example to describe the embodiment and each of the following embodiments.

[0056] Based on this, the application provides a signal lamp prompting method based on a wearable device, referring to Figure 1 , Figure 1 The flowchart of the first embodiment of the signal lamp prompting method based on the wearable device of the application is shown in the figure.

[0057] In the embodiment, the signal lamp prompting method based on the wearable device is applied to the control mainboard of the wearable device, the wearable device includes a device support, a control mainboard, a loudspeaker located inside the device support, a device frame body and a camera module located on the device frame body, the control mainboard is located inside the device support, and the method includes steps S10-S40:

[0058] Step S10, in response to a prompt instruction of a traffic signal lamp, determining a current prompt mode.

[0059] It should be noted that the wearable device includes a device support, a control mainboard, a loudspeaker, a device frame body, a camera module, an audio subboard, a battery and a contact interface. The device support and the device frame body constitute the device frame structure of the wearable device, and the device support and the device frame body are connected through an FPC (Flexible Printed Circuit) for the user to wear; the control mainboard, the audio subboard, the loudspeaker and the battery are located inside the device support; the contact interface is located on the bottom side of the device support; and the camera module is located above the device frame body. The control mainboard includes but is not limited to a main control SOC (System on a Chip), a PMIC (Power Management IC), an EMMC (Embedded Multi-Media Card) and a DRAM (Dynamic Random Access Memory).

[0060] It can be understood that the contact interface is a Pogo PIN interface in the embodiment, and the wearable device is charged through the Pogo PIN interface; in the audio sub-board, a Smart PA (Smart Power Amplifier) is configured for each speaker existing in the wearable device, automatic gain control is realized, and the clearest sound warning effect can be provided for the user; the wearable device can perform data interaction with the terminal device through Bluetooth or wireless WIFI, or perform wired data transmission with the terminal device through the Pogo PIN interface.

[0061] In a specific implementation, as the core of the wearable device, the selection of the camera module is also important, mainly including the following core parameters: for the Camera Sensor (camera sensor), the indicators such as the number of pixels, CFA type (Color Filter Array Type), power consumption, output frame rate, and pixel size are concerned; for the Camera Lens (camera lens), the indicators such as focusing distance and aperture are concerned.

[0062] It should be noted that, since the wearable device has the demand of operating the neural network, the SOC platform needs to provide sufficient computing power support. And for various application scenarios of the wearable device, such as the driving scenario, it is necessary to ensure that the photographing and response can be accurately performed in the high-speed driving state, which requires the platform architecture to have high processing capability. In addition, considering that the wearable device needs to perform data interaction with the terminal device, the platform needs to support the Bluetooth transmission protocol. Therefore, for the minimum system of the platform, the following requirements are required: sufficient computing power, for example, the application processor can be 4-core ARM Cortex-A53 and above, which can support NPU (Neural Processing Unit); support Bluetooth / Bluetooth Low Energy; support low-power state, the device can enter energy-saving mode when not in use to prolong the battery life; small package size; run Linux / Android operating system; 2GB and above of RAM (Random Access Memory); 16GB and above of EMMC.

[0063] It can be understood that, when the wearable device is in the form of glasses, such as Figure 2As shown, the device support is a right temple and a left temple, the device frame refers to a front frame of glasses, there is a speaker in each of the right temple and the left temple, the contact interface is located at the bottom side of any one of the temples, the audio sub-board, the battery and the control mainboard can be located inside any one of the temples, and the camera module is located at the upper side of the front frame of glasses. In this embodiment, in order to keep the weight of the glasses symmetrical, the control mainboard can be arranged inside the right temple, and the audio sub-board and the battery can be arranged inside the left temple, or other manners can exist, which are not limited in this embodiment. In this embodiment, the wearable device is taken as an example of glasses to illustrate the performance of the wearable device.

[0064] In a specific implementation, when the user wears the wearable device, the wearable device is sent a traffic signal lamp prompt instruction, and the traffic signal lamp prompt instruction includes a current prompt mode selected by the user. In this embodiment, the prompt mode includes but is not limited to a driver prompt mode and a pedestrian prompt mode. In the driver prompt mode, the user is currently driving a vehicle, and the color state of the signal lamp can be reminded by wearing the wearable device. In the pedestrian prompt mode, the user is currently walking or riding a non-motor vehicle, and the color state of the signal lamp can be reminded by wearing the wearable device. In this embodiment, the user can send the traffic signal lamp prompt instruction to the wearable device through a terminal device, click a control existing on the wearable device, voice or other forms, and the sending manner of the instruction is not limited in this embodiment.

[0065] It should be noted that when the wearable device receives the traffic signal lamp prompt instruction sent by the user, it indicates that the user expects to remind the color state of the traffic signal lamp through the wearable device. At this time, the wearable device determines the current prompt mode through the traffic signal lamp prompt instruction.

[0066] In a feasible implementation, before the step S10, the method can further include steps A11-A14.

[0067] Step A11, when receiving the traffic signal lamp prompt instruction sent by the user, acquiring a state detection image collected by the camera module.

[0068] It should be noted that when the traffic signal lamp prompt instruction sent by the user is received, the state detection image randomly collected by the camera module is used to detect whether the wearable device is correctly worn.

[0069] Step A12, determining a current device visual angle of the wearable device according to the state detection image.

[0070] Step A13, determining a current device state of the wearable device according to the current device visual angle.

[0071] Step A14, when the current device state is the normal wearing state, executing a step of determining the current prompt mode in response to a prompt instruction of a traffic signal light.

[0072] It should be noted that the feature detection algorithm, such as SIFT, SURF, ORB, etc. is used to detect the key points in the state detection image; the feature matching algorithm, such as FLANN, BFMatcher, etc. is used to match the key points in the state detection image, and the view angle change of the state detection image relative to the reference image is estimated through the feature matching result to obtain the current device view angle of the wearable device.

[0073] It can be understood that the current device view angle is compared with the view angle when the device is correctly worn, and whether the view angle change is within an acceptable range is determined by setting a reasonable threshold, and the current device state can be determined based on the determination result. In this embodiment, when the view angle change is less than the threshold, it is determined that the current device state is the normal wearing state; otherwise, it is the non-normal wearing state.

[0074] In a specific implementation, when the current device state is the normal wearing state, it indicates that the wearable device has been correctly worn by the user, and at this time, a step of determining the current prompt mode in response to a prompt instruction of a traffic signal light is executed.

[0075] Step S20, performing feature recognition on the current image frame collected by the camera module according to the image recognition model corresponding to the current prompt mode to obtain a signal light recognition result.

[0076] It should be noted that the image recognition model is a model for identifying whether the traffic signal light is included in the image frame, and the image recognition model can be obtained after training a model including a convolutional neural network and an application classifier; or it can be built based on a target detection algorithm or an image recognition algorithm, such as YOLO.

[0077] It can be understood that because the arrangement mode, signal light display shape and installation position of the traffic signal light are different under different prompt modes, and the traffic signal light corresponding to the pedestrian mode is the signal light for pedestrians / non-motor vehicles, and the traffic signal light corresponding to the driver mode is the signal light for motor vehicles. Therefore, different prompt modes correspond to different image recognition models, so as to avoid identifying the signal light for motor vehicles included in the current image frame in the driving mode. The image recognition model corresponding to the current prompt mode is called to perform feature recognition on the current image frame collected by the camera module to obtain the signal light recognition result of whether the traffic signal light is included in the current image frame.

[0078] In a feasible implementation, before the step S20, steps B11-B13 can also be included.

[0079] Step B11, upon receiving the lightweight identification model sent by the terminal, the performance of the lightweight identification model is detected.

[0080] Step B12, when the performance detection result is a preset qualified result, the lightweight identification model is deployed according to the operating system supported by the wearable device, and an image recognition model is obtained.

[0081] It should be noted that the training process of the image recognition model is performed on the terminal device. After the training is completed, the terminal device converts the target identification model obtained by training into a TensorFlow Lite model. Before conversion, the target identification model is a TensorFlow model. In this embodiment, the converted target identification model is the lightweight identification model, which is sent to the wearable device for model deployment.

[0082] It can be understood that when the wearable device receives the lightweight identification model, the performance of the lightweight identification model needs to be monitored to ensure that the accuracy and real-time performance of the model meet the expectations. If the performance detection result shows that the accuracy, delay, memory occupation and power consumption of the model all meet the preset standards or thresholds, it can be considered that the performance detection result of the model is a preset qualified result. At this time, the lightweight identification model is deployed through the operating system supported by the wearable device. The deployed lightweight identification model is the image recognition model.

[0083] In specific implementation, for ease of understanding, an example of model deployment is now given. For ease of use in Android / Linux devices, the TensorFlow model trained in the terminal device is converted into a TensorFlow Lite model. The TensorFlow Lite dependency is added in the Android APP of the wearable device or the TensorFlow Lite Runtime is installed in Linux. TensorFlow Lite API can be called to complete model deployment.

[0084] Step S30, when the signal lamp identification result is that there is a traffic signal lamp in the current image frame, the signal lamp state of the current image frame is determined.

[0085] It should be noted that when the signal lamp identification result is that there is a traffic signal lamp in the current image frame, the color of the traffic signal lamp in the current image frame is determined. In this embodiment, the signal lamp state refers to the current color of the traffic signal lamp. The signal lamp state can include red, green and yellow.

[0086] It can be understood that in the embodiment, the state of the signal light can be determined by the state recognition model, which is obtained by training a convolutional neural network through a large number of pictures labeled with the color of the signal light. When there is no traffic signal light in the current image frame, the image frame is discarded, and the next image frame is identified; when there is a traffic signal light in the current image frame, the current image frame is sent to the state recognition model for determination of the state of the signal light.

[0087] In a specific implementation, the deployment process of the state recognition model is consistent with the deployment process of the image recognition model, specifically: for the convenience of use in Android / Linux devices, the Tensorflow model trained in the terminal device is converted into a Tensorflow Lite model, and the Tensorflow Lite dependency is added in the Android APP of the wearable device or the Runtime of Tensorflow Lite is installed in Linux, so that the Tensorflow Lite API can be called to complete the model deployment.

[0088] Step S40, generating a signal light prompt signal according to the state of the signal light, and controlling the loudspeaker to broadcast the signal light prompt signal.

[0089] It should be noted that the corresponding signal light prompt signal is generated according to the state of the signal light, and the loudspeaker is controlled to broadcast the signal. In the embodiment, different signal light prompt signals correspond to different signal light states, for example, when the state of the signal light is red, the warning signal "red light ahead, please pay attention" is played; when the state of the signal light is green, the safety instruction signal "green light ahead, please pass" is played.

[0090] In a feasible implementation, the step S40 can include steps C11-C13.

[0091] Step C11, obtaining the broadcast warning sound effect selected by the user through the terminal.

[0092] Step C12, searching for the prompt sound effect corresponding to the state of the signal light in the broadcast warning sound effect.

[0093] Step C13, generating a signal light prompt signal according to the prompt sound effect, and controlling the loudspeaker to broadcast the signal light prompt signal.

[0094] It should be noted that in order to solve the problem of fixed single warning sound effect, a large number of different warning sound effects are stored in the terminal device, and the user can select the warning sound effect in the terminal, and the terminal device pushes the broadcast warning sound effect selected by the user to the wearable device. In each warning sound effect, different prompt sound effects are selected for different signal light states, so the wearable device needs to find the prompt sound effect corresponding to the signal light state in the broadcast warning sound effect selected by the user, and generate a signal light prompt signal based on the prompt sound effect, and control the loudspeaker to broadcast.

[0095] In a feasible implementation, the wearable device further comprises an audio sub-board located inside the device support; the step S40 can comprise steps D11-D13:

[0096] Step D11, generating a signal light prompt signal according to the signal light state.

[0097] Step D12, controlling the audio sub-board to perform audio processing on the signal light prompt signal to obtain a processed signal light prompt signal.

[0098] Step D13, controlling the loudspeaker to broadcast the processed signal light prompt signal according to a broadcast frequency.

[0099] It should be noted that in order to provide the user with the clearest sound warning effect, after generating the signal light prompt signal, the audio sub-board is controlled to perform filtering and other audio processing operations on the signal light prompt signal, thereby obtaining a processed signal light prompt signal, and the loudspeaker is controlled to broadcast it according to the set broadcast frequency. In this embodiment, the broadcast frequency can be set to broadcast once every 10s, or other frequencies, which are not limited in this embodiment.

[0100] The embodiment provides a traffic light prompting method based on a wearable device. The traffic light prompting method based on the wearable device is applied to a control mainboard of the wearable device. The wearable device comprises a device support, a control mainboard, a loudspeaker located in the interior of the device support, a device frame body, and a camera module located on the device frame body. The control mainboard is located in the interior of the device support. The method comprises the following steps: in response to a prompting instruction of a traffic light, determining a current prompting mode; performing feature recognition on a current image frame collected by the camera module according to an image recognition model corresponding to the current prompting mode, to obtain a traffic light recognition result; when the traffic light recognition result is that the current image frame contains a traffic light, determining a traffic light state of the current image frame; generating a traffic light prompting signal according to the traffic light state, and controlling the loudspeaker to broadcast the traffic light prompting signal. In this way, the state of the traffic light can be accurately recognized by the wearable device, and a corresponding prompting signal can be generated and broadcasted to prompt the driver or the pedestrian, so that the situation of running a red light is avoided, the occurrence rate of traffic accidents is further reduced, and the safety of the user is ensured.

[0101] Based on the first embodiment of the present application, the same or similar contents as the above-mentioned first embodiment can be referred to the above description, and will not be described hereinafter. On this basis, please refer to Figure 3 , step S20, the traffic light prompting method based on the wearable device further comprises steps S21-S22:

[0102] Step S21, when the current prompting mode is a pedestrian prompting mode, a first recognition model is called to perform feature recognition on the current image frame collected by the camera module, to obtain a traffic light recognition result. The first recognition model is obtained by training a target detection model by a plurality of first training images, and then performing lightweight processing on the trained target detection model. Each first training image is obtained by labeling signal light arrangement information and signal light display forms in each sample signal light image under different environments.

[0103] It should be noted that the first recognition model is an image recognition model corresponding to the pedestrian prompting mode. When the current prompting mode is the pedestrian prompting mode, the first recognition model is called to perform feature recognition on the current image frame, to output a traffic light recognition result indicating whether there is a traffic light in the current image frame.

[0104] It can be understood that due to the arrangement of traffic signal lights, the shape of signal light display and the installation position are various, and the size of signal light is also various, and the environment of signal light is also various with the weather. Therefore, the terminal needs to collect a plurality of sample signal light images under various weather environments, and under different weather environments, different arrangement modes, display shapes, installation positions and signal light sizes are corresponded to sample signal light images, and meanwhile, the shooting angles of signal lights in sample signal light images also exist differences, that is, signal light data of left, middle and right angles also needs to be shot. In addition, the traffic signal light contained in the sample signal light image is the traffic signal light corresponding to the pedestrian.

[0105] In a specific implementation, the signal light display form includes a signal light display shape and a signal light size; and the signal light arrangement information includes a signal light arrangement mode and a signal light installation position. The signal light arrangement information and the signal light display form in each sample signal light image under different environments are labeled with features to obtain first training images under different environments.

[0106] It should be noted that the target detection model can be an untrained model built by a target detection algorithm, and the terminal device trains the target detection model through a plurality of first training images, and sends the trained target detection model to the wearable device after lightening processing, so as to obtain a deployed first recognition model.

[0107] Step S22, when the current prompt mode is the driver prompt mode, a second recognition model is called to perform feature recognition on the current image frame collected by the camera module to obtain a signal light recognition result.

[0108] It should be noted that the second recognition model is an image recognition model corresponding to the driver prompt mode, and when the current prompt mode is the driver prompt mode, the second recognition model is called to perform feature recognition on the current image frame, and a signal light recognition result of whether there is a traffic signal light in the current image frame is output. In this embodiment, the training and deployment process of the second recognition model is consistent with the training and deployment process of the first recognition model, and the traffic signal light contained in the image involved in the training process is the traffic signal light corresponding to the motor vehicle. In this embodiment, the first recognition model and the second recognition model both have good robustness.

[0109] In a feasible implementation, step S30 can further include steps E11-E12:

[0110] Step E11, when the signal lamp recognition result is that there is a traffic signal lamp in the current image frame, input the current image frame to a state recognition model, the state recognition model is obtained by training a convolutional neural network by the terminal through a plurality of second training images, and the trained convolutional neural network is processed to be lightweight, and each second training image is obtained by labeling the color state of the signal lamp in each sample signal lamp image in different environments.

[0111] Step E12, according to the color state recognition of the current image frame by the state recognition model, determine the signal lamp state of the current image frame.

[0112] It should be noted that when the signal lamp recognition result is that there is a traffic signal lamp in the current image frame, the current image frame is input to the state recognition model, and the state recognition model outputs the signal lamp state of the current image frame.

[0113] It can be understood that the terminal device collects each sample signal lamp image in different weather environments, and labels the color state of the signal lamp of each sample signal lamp image to obtain a large number of second training images in different environments. Through a large number of second training images, the convolutional neural network is trained, and the trained convolutional neural network is processed to be lightweight and sent to the wearable device for model deployment, so as to obtain the deployed state recognition model.

[0114] In a specific implementation, through the back propagation algorithm, the state recognition model can adjust the weight to optimize the prediction result. In this embodiment, the Tensorflow deep learning software library is used for convolutional neural network operation, and a large number of second training images are divided into a training set and a test set, and the neural network is trained, so as to obtain the state recognition model deployed in the wearable device.

[0115] In this embodiment, when the signal lamp recognition result is that there is a traffic signal lamp in the current image frame, the current image frame is input to a state recognition model, the state recognition model is obtained by training a convolutional neural network by the terminal through a plurality of second training images, and the trained convolutional neural network is processed to be lightweight, and each second training image is obtained by labeling the color state of the signal lamp in each sample signal lamp image in different environments; according to the color state recognition of the current image frame by the state recognition model, determine the signal lamp state of the current image frame. Through the above-mentioned manner, the state recognition model can accurately obtain the signal lamp state.

[0116] The embodiment provides a traffic light prompting method based on a wearable device, and the traffic light prompting method comprises the following steps: when the current prompting mode is a pedestrian prompting mode, calling a first recognition model to perform feature recognition on a current image frame collected by a camera module to obtain a traffic light recognition result, wherein the first recognition model is obtained by training a target detection model by a terminal through a plurality of first training images, and the target detection model is obtained after being lightened, and each first training image is obtained after labeling signal light arrangement information and signal light display forms in each sample signal light image in different environments; when the current prompting mode is a driver prompting mode, calling a second recognition model to perform feature recognition on the current image frame collected by the camera module to obtain the traffic light recognition result. Through the above method, the traffic light recognition result in different scenes can be accurately obtained based on different image recognition models.

[0117] Exemplarily, in order to help understand the implementation process of the traffic light prompting method based on the wearable device obtained after the above embodiment one, please refer to Figure 4 , Figure 4 A brief flowchart of a traffic light prompting method based on a wearable device is provided, and specifically:

[0118] In the embodiment, the performance form of the wearable device is taken as smart glasses, the camera module photographs pictures at regular time intervals in the driver prompting mode or the pedestrian mode, first, the image recognition model is used for traffic light recognition, if there is no traffic light, the frame image data is discarded. If there is a traffic light, the frame data is sent to a state recognition model to judge the color of the traffic light, and finally, the red light warning sound effect or the green light safety indication sound effect is played on the glasses end main app, so that the further warning effect is achieved.

[0119] It should be noted that the above examples are only used for understanding the present application, and do not constitute a limitation on the traffic light prompting method based on the wearable device of the present application, and more forms of simple transformation based on the technical concept are within the protection scope of the present application.

[0120] The present application also provides a traffic light prompting device based on a wearable device, please refer to Figure 5 , the traffic light prompting device based on the wearable device comprises:

[0121] The processing module 10 is configured to determine a current prompting mode in response to a prompting instruction of a traffic light.

[0122] The recognition module 20 is configured to perform feature recognition on a current image frame collected by the camera module according to an image recognition model corresponding to the current prompting mode to obtain a traffic light recognition result.

[0123] The processing module 10 is configured to determine a signal lamp state of the current image frame when the signal lamp recognition result indicates that there is a traffic signal lamp in the current image frame.

[0124] The control module 30 is configured to generate a signal lamp prompt signal according to the signal lamp state, and control the loudspeaker to broadcast the signal lamp prompt signal.

[0125] Optionally, the identification module 20 is further configured to:

[0126] When the current prompt mode is a pedestrian prompt mode, a first identification model is called to perform feature recognition on the current image frame collected by the camera module to obtain a signal lamp recognition result, the first identification model being obtained by training a target detection model by a terminal through a plurality of first training images, and performing lightweight processing on the trained target detection model, each first training image being obtained by labeling signal lamp arrangement information and signal lamp display forms in each sample signal lamp image in different environments; when the current prompt mode is a driver prompt mode, a second identification model is called to perform feature recognition on the current image frame collected by the camera module to obtain a signal lamp recognition result.

[0127] Optionally, the processing module 10 is further configured to:

[0128] When the signal lamp recognition result indicates that there is a traffic signal lamp in the current image frame, the current image frame is input to a state recognition model, the state recognition model being obtained by training a convolutional neural network by a terminal through a plurality of second training images, and performing lightweight processing on the trained convolutional neural network, each second training image being obtained by labeling signal lamp color states in each sample signal lamp image in different environments; and the state recognition model is used to perform color state recognition on the current image frame to determine the signal lamp state of the current image frame.

[0129] Optionally, the control module 30 is further configured to:

[0130] obtain a broadcast warning sound effect selected by a user through the terminal; search for a prompt sound effect corresponding to the signal lamp state in the broadcast warning sound effect; generate a signal lamp prompt signal according to the prompt sound effect, and control the loudspeaker to broadcast the signal lamp prompt signal.

[0131] Optionally, the control module 30 is further configured to:

[0132] generate a signal lamp prompt signal according to the signal lamp state; control the audio sub-board to perform audio processing on the signal lamp prompt signal to obtain a processed signal lamp prompt signal; and control the loudspeaker to broadcast the processed signal lamp prompt signal at a broadcast frequency.

[0133] Optionally, the processing module 10 is further configured to:

[0134] In response to receiving the prompt instruction of the traffic signal light sent by the user, a state detection image collected by the camera module is acquired; a current device perspective of the wearable device is determined according to the state detection image; a current device state of the wearable device is determined according to the current device perspective; and when the current device state is a normal wearing state, the step of determining a current prompt mode in response to the prompt instruction of the traffic signal light is performed.

[0135] Optionally, the identification module 20 is further configured to:

[0136] In response to receiving the lightweight identification model sent by the terminal, the performance of the lightweight identification model is detected; and when the performance detection result is a preset qualified result, the lightweight identification model is deployed according to an operating system supported by the wearable device to obtain an image identification model.

[0137] The signal light prompting device based on the wearable device provided in the application adopts the signal light prompting method based on the wearable device in the above embodiment, and can solve the technical problem that the user cannot be prompted by the traffic signal light in the prior art. Compared with the prior art, the signal light prompting device based on the wearable device provided in the application has the same beneficial effects as the signal light prompting method based on the wearable device provided in the above embodiment, and other technical features in the signal light prompting device based on the wearable device are the same as the features disclosed in the above embodiment method, and thus will not be described here.

[0138] The application provides a wearable device, which comprises at least one processor and a memory in communication connection with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the signal light prompting method based on the wearable device in the above embodiment one.

[0139] Reference will now be made to the following description Figure 6 which shows a structural schematic diagram of a wearable device suitable for implementing the embodiments of the application. The wearable device in the embodiments of the application can include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (Personal Digital Assistant), PADs (Portable Application Description), PMPs (Portable Media Player), vehicle-mounted terminals (for example, vehicle-mounted navigation terminals), and the like, and fixed terminals such as digital TVs, desktop computers, and the like.Figure 6 The wearable device shown is merely an example and should not impose any limitation on the function and use range of the embodiments of the present application.

[0140] As shown in Figure 6 The wearable device can include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to a program stored in a read only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. In the RAM 1004, various programs and data required for the operation of the wearable device are also stored. The processing device 1001, the ROM 1002, and the RAM 1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. In general, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; the storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the wearable device to communicate wirelessly or wired with other devices to exchange data. Although the wearable device having various systems is shown in the figure, it should be understood that all the systems shown are not required to be implemented or possessed. More or less systems can be alternatively implemented or possessed.

[0141] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product including a computer program carried on a computer readable medium, the computer program containing program codes for executing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network through the communication device, or installed from the storage device 1003, or installed from the ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the methods of the embodiments disclosed in the present application are performed.

[0142] The wearable device provided by the present application adopts the signal lamp prompting method based on the wearable device in the above embodiment, and can solve the technical problem that the user cannot be prompted about the traffic signal lamp in the prior art. Compared with the prior art, the wearable device provided by the present application has the same beneficial effects as the signal lamp prompting method based on the wearable device provided by the above embodiment, and other technical features in the wearable device are the same as the features disclosed in the previous embodiment method, which will not be described here.

[0143] It should be understood that parts of the present application can be realized by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0144] The above describes only the specific implementation of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0145] The present application provides a computer readable storage medium having computer readable program instructions (i.e. computer programs) stored thereon, the computer readable program instructions being used to execute the signal lamp prompting method based on the wearable device in the above embodiment.

[0146] The computer readable storage medium provided by the present application may, for example, be a U disk, but is not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, system or device, or any combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection with one or more conductive wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present embodiment, the computer readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system or device. The program code contained on the computer readable storage medium can be transmitted by any appropriate medium, including but not limited to: electrical wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0147] The computer readable storage medium can be included in the wearable device, or can exist separately from the wearable device.

[0148] The computer readable storage medium carries one or more programs, which, when executed by the wearable device, cause the wearable device to: determine a current indication mode in response to an indication instruction of a traffic signal light; perform feature recognition on a current image frame collected by the camera module according to an image recognition model corresponding to the current indication mode, to obtain a signal light recognition result; determine a signal light state of the current image frame when the signal light recognition result indicates that the current image frame contains a traffic signal light; generate a signal light indication signal according to the signal light state, and control the loudspeaker to broadcast the signal light indication signal.

[0149] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0150] The flow and block diagrams in the drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flow and block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart illustrations, and combinations thereof, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or combinations of hardware and software.

[0151] The modules involved in the embodiments of the present application can be implemented in the form of software or in the form of hardware. In some cases, the name of the module does not constitute a limitation on the unit itself.

[0152] The readable storage medium provided by the present application is a computer readable storage medium, which stores computer readable program instructions (i.e. computer program) for executing the above-mentioned wearable device-based traffic light prompting method, and can solve the technical problem that the user cannot be prompted by the traffic light in the prior art. Compared with the prior art, the computer readable storage medium provided by the present application has the same beneficial effects as the wearable device-based traffic light prompting method provided by the above-mentioned embodiments, which will not be repeated here.

[0153] The present application also provides a computer program product comprising a computer program, which, when executed by a processor, implements the steps of the above-mentioned wearable device-based traffic light prompting method.

[0154] The computer program product provided by the present application can solve the technical problem that the user cannot be prompted by the traffic light in the prior art. Compared with the prior art, the computer program product provided by the present application has the same beneficial effects as the wearable device-based traffic light prompting method provided by the above-mentioned embodiments, which will not be repeated here.

[0155] The above only describes some embodiments of the present application, and does not limit the patent scope of the present application, and any equivalent structural transformation made by using the content of the present application specification and drawings, or direct / indirect application in other related technical fields is included in the patent protection scope of the present application.

Claims

1. A traffic light prompting method based on a wearable device, characterized by, The method is applied to a control mainboard of the wearable device, the wearable device comprising a device support, a control mainboard, a loudspeaker located inside the device support, a device frame body, and a camera module located on the device frame body, the control mainboard being located inside the device support; The method comprises: determining a current prompt mode in response to a prompt instruction of a traffic signal light; performing feature recognition on a current image frame collected by the camera module according to an image recognition model corresponding to the current prompt mode to obtain a signal light recognition result; when the signal light recognition result is that there is a traffic signal light in the current image frame, determining a signal light state of the current image frame; generating a signal light prompt signal according to the signal light state and controlling the loudspeaker to broadcast the signal light prompt signal.

2. The method of claim 1, wherein, The step of performing feature recognition on a current image frame collected by the camera module according to an image recognition model corresponding to the current prompt mode to obtain a signal light recognition result comprises: when the current prompt mode is a pedestrian prompt mode, calling a first recognition model to perform feature recognition on the current image frame collected by the camera module to obtain a signal light recognition result, the first recognition model being obtained by training a target detection model by a terminal through a plurality of first training images and performing lightweight processing on the trained target detection model, each first training image being obtained by labeling signal light arrangement information and signal light display forms in each sample signal light image under different environments; when the current prompt mode is a driver prompt mode, calling a second recognition model to perform feature recognition on the current image frame collected by the camera module to obtain a signal light recognition result.

3. The method of claim 1, wherein, The step of determining a signal light state of the current image frame when the signal light recognition result is that there is a traffic signal light in the current image frame comprises: when the signal light recognition result is that there is a traffic signal light in the current image frame, inputting the current image frame into a state recognition model, the state recognition model being obtained by training a convolutional neural network by a terminal through a plurality of second training images and performing lightweight processing on the trained convolutional neural network, each second training image being obtained by labeling signal light color states in each sample signal light image under different environments; performing color state recognition on the current image frame according to the state recognition model to determine the signal light state of the current image frame.

4. The method of claim 1, wherein, The step of generating a signal light prompt signal according to the signal light state and controlling the loudspeaker to broadcast the signal light prompt signal comprises: obtaining a broadcast warning sound effect selected by a user through a terminal; finding a prompt sound effect corresponding to the signal light state in the broadcast warning sound effect; generating a signal light prompt signal according to the prompt sound effect and controlling the loudspeaker to broadcast the signal light prompt signal.

5. The method of claim 1, wherein, The wearable device further comprises an audio subboard located inside the device support; The step of generating a signal light prompt signal according to the signal light state and controlling the loudspeaker to broadcast the signal light prompt signal comprises: generate a signal lamp prompt signal according to the signal lamp state; control the audio sub-board to perform audio processing on the signal lamp prompt signal to obtain a processed signal lamp prompt signal; control the loudspeaker to broadcast the processed signal lamp prompt signal according to a broadcast frequency.

6. The method of any one of claims 1 to 5, wherein, Before the step of determining the current prompt mode in response to the prompt instruction of the traffic signal lamp, the method further includes: when receiving the prompt instruction of the traffic signal lamp sent by the user, obtaining a state detection image collected by the camera module; determining a current device visual angle of the wearable device according to the state detection image; determining a current device state of the wearable device according to the current device visual angle; and when the current device state is a normal wearing state, performing the step of determining the current prompt mode in response to the prompt instruction of the traffic signal lamp.

7. The method of any one of claims 1 to 5, wherein, Before the step of performing feature recognition on a current image frame collected by the camera module according to an image recognition model corresponding to the current prompt mode to obtain a signal lamp recognition result, the method further includes: when receiving a lightweight recognition model sent by the terminal, performing performance detection on the lightweight recognition model; when a performance detection result is a preset qualified result, deploying the lightweight recognition model according to an operating system supported by the wearable device to obtain the image recognition model.

8. A wearable device based traffic light cueing apparatus, characterized in that, The signal lamp prompting device based on the wearable device includes: a processing module configured to determine a current prompt mode in response to a prompt instruction of a traffic signal lamp; an identification module configured to perform feature recognition on a current image frame collected by the camera module according to an image recognition model corresponding to the current prompt mode to obtain a signal lamp recognition result; the processing module is configured to determine a signal lamp state of the current image frame when the signal lamp recognition result indicates that the traffic signal lamp exists in the current image frame; a control module configured to generate a signal lamp prompt signal according to the signal lamp state and control the loudspeaker to broadcast the signal lamp prompt signal.

9. A wearable device, comprising: The wearable device includes a memory, a processor, and a signal lamp prompting program based on the wearable device stored on the memory and executable on the processor, and the signal lamp prompting program based on the wearable device is configured to implement the signal lamp prompting method based on the wearable device.

10. A storage medium, characterized by The storage medium stores a signal lamp prompting program based on the wearable device, and the signal lamp prompting program based on the wearable device implements the signal lamp prompting method based on the wearable device when executed by the processor.

Images