Traffic signal light perception method, device control method, device, and medium
By extracting and detecting features from environmental images using a traffic light perception model and combining it with an attention mechanism to acquire virtual lights, the problem of low error tolerance and poor generalization ability of traditional traffic light perception methods is solved, enabling end-to-end acquisition of traffic light status and safe driving of intelligent devices.
Patent Information
- Application Number
- CN202411903986.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-20
- Publication Date
- 2026-06-23
AI Technical Summary
In existing technologies, traffic light sensing methods are sensitive to detection results, have low error tolerance, poor post-processing generalization ability, and are difficult to upgrade through data-driven iterations, resulting in inaccurate traffic light status perception.
A traffic light perception model is used to process environmental images. Virtual lights are obtained through feature extraction and detection networks. An attention mechanism is combined to achieve end-to-end acquisition of traffic direction and status. Iterative training is performed using environmental image samples and their annotation information.
It enables direct mapping from environmental images to virtual lights, improves the generalization ability and accuracy of traffic light perception, supports iterative upgrades of traffic light perception methods, and ensures safe driving of intelligent devices at traffic light intersections.
Smart Images
Figure CN122265964A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of autonomous driving technology, specifically to a method for sensing traffic lights, a method for controlling devices, devices, and media. Background Technology
[0002] When controlling a vehicle for autonomous driving at an intersection, it is necessary to accurately obtain the status of the traffic direction indicated by the traffic lights, and then determine the vehicle's driving decision based on the status. For example, if the vehicle's planned trajectory is to go straight within the intersection, but the status of the straight direction is prohibited, then the vehicle needs to be controlled to stop at the stop line before the intersection, wait for the status to change to allow passage, and then control the vehicle to pass through the intersection.
[0003] Currently, a combination of perception models and post-processing can be used to perceive traffic lights and obtain their state. Specifically, a perception model is first used to perceive the position of each individual light frame and its attributes (including light head shape, color, and countdown numbers) from the traffic light image. Then, a rule-based method is used to post-process the position and attributes of the light frames to obtain the state of each traffic direction. In the post-processing, the light frames are first tracked and smoothed temporally to obtain their accurate position and attributes. Then, candidate light frames located in or near the road in front of the vehicle are selected based on rules such as distance and angle. Finally, according to the correspondence rules between light head shape and traffic direction, the traffic direction corresponding to the light head shape of the candidate light frames is obtained, and the state of the traffic direction (such as color, countdown numbers, and flashing status) is determined based on the attributes of the candidate light frames.
[0004] In this method, the perception model only needs to perform light frame detection and attribute classification on individual lights in the image, without needing to focus on the features of the intersection where the traffic light is located, which reduces the training difficulty of the perception model. However, the post-processing stage has the following disadvantages: (1) It is too sensitive to the detection results of the detection model and has a low error tolerance. If the detection results fluctuate, the post-processing stage may not be able to obtain the state of the traffic light. (2) Since it is impossible to cover all traffic light scenarios through rules, the generalization ability of the post-processing is poor and the upper limit is low. (3) Although the detection model can be iteratively upgraded through data-driven means, since the post-processing adopts a rule-based method, it is difficult to iteratively upgrade the entire method through data-driven means to improve the perception ability of traffic lights.
[0005] Accordingly, a new technical solution is needed in this field to solve the above problems. Summary of the Invention
[0006] In order to overcome the above-mentioned deficiencies, this application is made to solve or at least partially solve the following technical problem: how to accurately sense the status of traffic lights.
[0007] In a first aspect, a method for sensing traffic lights is provided, the method comprising:
[0008] Acquire environmental images collected by the smart device, wherein the environmental images are images of the environment in which the smart device is located, and the environmental images include traffic lights;
[0009] The environmental image is input into the traffic light perception model for processing to obtain the virtual light corresponding to the environmental image;
[0010] The virtual lights are used for driving control of the smart device. The virtual lights include multiple traffic direction indicators and the indication status of each traffic direction indicator. The traffic direction indicators are used to indicate the driving direction of the smart device.
[0011] The traffic light perception model is trained based on samples of the environmental image and their annotation information, the annotation information including the virtual light corresponding to the sample.
[0012] In one technical solution of the above-mentioned traffic light sensing method, the traffic light sensing model includes a traffic light sensing module and a virtual light acquisition module;
[0013] The traffic light sensing module is configured to: perform traffic light sensing on the environmental image to obtain the traffic light features of the traffic lights in the environmental image;
[0014] The virtual light acquisition module is configured to acquire the virtual light corresponding to the environmental image based on the traffic light characteristics.
[0015] In one technical solution of the above-mentioned traffic light sensing method, the traffic light features include the illuminated light frame and the attributes of the illuminated light frame, wherein there is at least one illuminated lamp head within the illuminated light frame, and the attributes include the type, color, and countdown digits of the lamp head.
[0016] In one technical solution of the above-mentioned traffic light sensing method, the traffic light sensing module includes a feature extraction network and a traffic light detection network;
[0017] The feature extraction network is configured to: extract features from the environmental image to obtain image features of the environmental image from a bird's-eye view;
[0018] The traffic light detection network is configured to detect traffic lights from the image features in the bird's-eye view to obtain the traffic light features.
[0019] In one technical solution of the above-mentioned traffic light perception method, the virtual light acquisition module is configured to acquire the virtual light based on the image features and the traffic light features.
[0020] In one technical solution of the above-mentioned traffic light perception method, obtaining the virtual light based on the image features and the traffic light features includes:
[0021] Based on the model parameters of the traffic light perception model, the query features of traffic direction in the attention mechanism are obtained, wherein the traffic direction is the traffic direction in the virtual light;
[0022] Based on the query features, cross-attention processing is performed on the traffic direction indication and the image features to obtain a first processing result;
[0023] The first processing result is cross-attention processing is performed on the traffic light features to obtain a second processing result, which includes the indication status of the traffic direction.
[0024] The virtual light is obtained based on the traffic direction and its status.
[0025] In one technical solution of the above-mentioned traffic signal light sensing method, the indication state of the traffic direction includes color, flashing state, and countdown numbers.
[0026] In a second aspect, a control method for a smart device is provided, the method comprising:
[0027] Using any of the traffic light sensing methods provided in the first aspect, the virtual lights corresponding to the traffic lights in the environment where the smart device is located are obtained;
[0028] Obtain the driving direction of the intelligent device;
[0029] Based on the driving direction, query the indication status of the target direction from the virtual lights, where the target direction is the traffic indication direction used to indicate the driving direction;
[0030] The intelligent device is controlled for driving based on the indicated state.
[0031] In a third aspect, a smart device is provided, the smart device including at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program, which, when executed by the at least one processor, implements the method described in any of the technical solutions provided in the first or second aspect.
[0032] In a third aspect, a computer-readable storage medium is provided, wherein a plurality of program codes are stored therein, the program codes being adapted to be loaded and executed by a processor to perform the method described in any of the technical solutions provided in the first or second aspect above.
[0033] The above-described technical solutions of this application have at least one or more of the following beneficial effects:
[0034] In one technical solution of the traffic light perception method provided in this application, the method can acquire an environmental image collected by a smart device. The environmental image is an image of the environment in which the smart device is located, and the environmental image includes traffic lights. The environmental image is input into a traffic light perception model for processing to obtain a virtual light corresponding to the environmental image. The virtual light is used by the smart device for driving control. The virtual light includes multiple traffic direction indicators and the indication status of each traffic direction indicator. The traffic direction indicators are used to indicate the driving direction of the smart device. The traffic light perception model is trained based on samples of the environmental image and their annotation information. The annotation information includes the virtual light corresponding to the sample.
[0035] In the above implementation scheme, the traffic light perception model can directly map or obtain virtual lights from environmental images collected by smart devices. These virtual lights include multiple traffic directions and their respective indication states, effectively establishing a direct mapping relationship between the input (environmental image) and the output (traffic directions and their indication states), achieving end-to-end acquisition of traffic directions and their indication states. Furthermore, for any environment, the traffic direction and its indication state can be directly and accurately obtained from the environmental image through the traffic light perception model, significantly improving the generalization ability of traffic light perception. Moreover, iterative training of the traffic light perception model using environmental image samples and their annotation information can continuously improve the model's perception capability, achieving iterative upgrades to the entire traffic light perception method—that is, realizing iterative upgrades of the traffic light perception method through a data-driven approach. Attached Figure Description
[0036] The disclosure of this application will become more readily understood with reference to the accompanying drawings. It will be readily understood by those skilled in the art that these drawings are for illustrative purposes only and are not intended to limit the scope of protection of this application. Wherein:
[0037] Figure 1 This is a schematic flowchart of the main steps of a traffic signal light sensing method according to an embodiment of this application;
[0038] Figure 2 This is a schematic diagram of the main structure of a traffic light sensing model according to an embodiment of this application;
[0039] Figure 3 This is a schematic diagram of the main process of existing traffic light sensing methods;
[0040] Figure 4 This is a schematic diagram of the main flow of a traffic signal light sensing method according to an embodiment of this application;
[0041] Figure 5 This is a schematic flowchart of the main steps of a control method for a smart device according to an embodiment of this application;
[0042] Figure 6 This is a schematic diagram of the main structure of a smart device according to an embodiment of this application.
[0043] Figure label:
[0044] 11: Memory; 12: Processor. Detailed Implementation
[0045] Some embodiments of this application are described below with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of this application and are not intended to limit the scope of protection of this application.
[0046] In the description of this application, "processor" can include hardware, software, or a combination of both. A processor can be a central processing unit, microprocessor, graphics processor, digital signal processor, or any other suitable processor. A processor has data and / or signal processing capabilities. A processor can be implemented in software, in hardware, or a combination of both. Computer-readable storage media includes any suitable medium capable of storing program code, such as magnetic disks, hard disks, optical disks, flash memory, read-only memory, random access memory, etc.
[0047] The relevant user personal information that may be involved in the various embodiments of this application is processed in strict accordance with the requirements of laws and regulations, following the principles of legality, legitimacy, and necessity, based on the reasonable purpose of the business scenario, and includes personal information that users actively provide or that is generated as a result of using the product / service, as well as personal information obtained with user authorization.
[0048] The personal information processed in this application will vary depending on the specific product / service scenario and will be based on the specific scenario in which the user uses the product / service. This may involve the user's account information, device information, driving information, vehicle information, or other related information. This application will treat the user's personal information and its processing with the utmost diligence.
[0049] This application attaches great importance to the security of users' personal information and has taken reasonable and feasible security protection measures that comply with industry standards to protect users' information and prevent unauthorized access, disclosure, use, modification, damage or loss of personal information.
[0050] The following describes an embodiment of the traffic light sensing method provided in this application, which is executed by a smart device.
[0051] See appendix Figure 1 , Figure 1 This is a schematic flowchart illustrating the main steps of a traffic light sensing method according to an embodiment of this application. Figure 1 As shown, the traffic light sensing method in this embodiment mainly includes the following steps S101 to S102.
[0052] Step S101: Acquire the environmental image collected by the smart device. The environmental image is an image of the environment in which the smart device is located, and the environmental image includes traffic lights.
[0053] Intelligent devices can include driving equipment, intelligent vehicles, robots, and other similar devices. The environment can be, but is not limited to, scenarios where intelligent devices are driving or being tested. Furthermore, traffic lights can be traffic lights that appear at intersections, tunnels, toll booths, main lanes, and other similar locations.
[0054] Smart devices are equipped with at least one camera, which can be used to capture images of the environment.
[0055] Step S102: Input the environmental image into the traffic light perception model for processing to obtain the virtual light corresponding to the environmental image. The traffic light perception model can be deployed on a smart device.
[0056] Virtual lights are used for driving control of smart devices. Virtual lights include multiple traffic directions and the indication status of each traffic direction. The traffic directions are used to indicate the driving direction of the smart device.
[0057] Traffic direction instructions can include going straight, turning left, turning right, and making a U-turn.
[0058] Traffic direction indication status can include color, flashing status, and countdown timer. Color refers to the target color used for traffic guidance; target colors can include green, red, and yellow. Green indicates permission to proceed, red indicates prohibition, and yellow indicates a warning. Flashing status can be either flashing or non-flashing. Flashing indicates that the traffic direction color alternates repeatedly between the current target color and black. The countdown timer indicates the remaining time the traffic direction color will remain the current target color; after the remaining time reaches 0, it will change to another target color.
[0059] The following is an explanation of the traffic light perception model.
[0060] The traffic light perception model is a deep learning model, which can be a neural network. In some implementations, the deep learning model can be a deep neural network or a convolutional neural network (CNN), or it can be a recurrent neural network (RNN) or a transformer, etc.
[0061] The traffic light perception model is trained based on environmental image samples and their annotation information. The annotation information includes the virtual light corresponding to the sample. The environmental image samples have the same meaning as the environmental images in step S101 above, the main difference being that the environmental image samples are the environmental images used in the model training phase, while the environmental images in step S101 are the images used in the model application phase. Similarly, the virtual lights in the annotation information have the same meaning as the virtual lights in step S101 above.
[0062] The traffic light perception model is trained in a supervised manner using samples and annotation information of environmental images. This allows the trained model to learn the ability to extract virtual lights from environmental images. In other words, the traffic light perception model can directly map or obtain virtual lights from environmental images, establishing a direct mapping relationship between input (environmental image) and output (virtual light), achieving end-to-end acquisition of virtual lights. The traffic light perception model can be understood as an end-to-end model. In this embodiment, conventional supervised training methods can be used to train the traffic light perception model; this embodiment does not impose specific limitations on this approach.
[0063] Based on the methods described in steps S101 to S102 above, for traffic lights in any environment, the traffic light perception model can directly and accurately obtain the traffic direction indication status (used to indicate the driving direction of intelligent devices when driving at the intersection where the traffic light is located) in the virtual light from the environmental image. Furthermore, iterative training of the traffic light perception model can continuously improve its perception capability, achieving iterative upgrades to the entire traffic light perception method.
[0064] The following describes an embodiment of the traffic light sensing method provided in this application, specifically the traffic light sensing model.
[0065] In some embodiments of this application, the traffic light perception model includes a traffic light perception module and a virtual light acquisition module. The traffic light perception module can be configured to perceive traffic lights in an environmental image and obtain the traffic light features of the traffic lights in the environmental image. The virtual light acquisition module can be configured to acquire the virtual light corresponding to the environmental image based on the traffic light features.
[0066] As described in the foregoing embodiments, the traffic light perception model is trained based on samples of environmental images and their annotation information. The annotation information may include the virtual lights corresponding to the samples. In some implementations, traffic light features from the samples can be added to the annotation information. Simultaneously, the traffic light perception model is trained under supervised supervision using both the virtual lights and the traffic light features in the annotation information. This allows the model to learn not only how to obtain virtual lights from images but also how to accurately detect traffic light features from images. Furthermore, these traffic light features can further improve the accuracy of the virtual lights. In other words, supervised training of traffic light features can assist in the supervised training of virtual lights, significantly improving the model's convergence speed and quickly obtaining a traffic light perception model capable of accurately perceiving virtual lights.
[0067] In some implementations, traffic light features include an illuminated frame and attributes of the illuminated frame. A frame of a traffic light contains one or more light heads. A light head is considered illuminated if its color is a target color used for traffic guidance (such as green, red, or yellow). Furthermore, a frame is considered illuminated if at least one light head is illuminated. The attributes of an illuminated frame include the type, color, and countdown timer of the illuminated light head.
[0068] The type of light head that illuminates can also be understood as the shape of the light head. Based on the semantic meaning of the light head shape, light heads can include turn signals and countdown lights. The semantic meaning of the turn signal light head shape is the traffic direction indication. Turn signal light head shapes can include discs, straight arrows, left turn arrows, right turn arrows, and U-turn arrows. Different light head shapes can represent different traffic directions. For example, a disc indicates straight, left turn, and U-turn; a straight arrow indicates straight ahead. The light head shape of a countdown light is a number. The number indicates the remaining time the turn signal will display the current target color. After the remaining time reaches 0, the turn signal will display another target color.
[0069] The following description continues with an embodiment of the traffic light sensing method provided in this application, still focusing on the traffic light sensing model.
[0070] See appendix Figure 2 In some embodiments of this application, the traffic light perception module may include a feature extraction network and a traffic light detection network. The feature extraction network may be configured to extract features from the environmental image to obtain image features of the environmental image from a bird's-eye view perspective. The traffic light detection network may be configured to detect traffic lights from the image features from the bird's-eye view perspective to obtain traffic light features. Furthermore, the virtual light acquisition module may be configured to acquire virtual lights based on the image features and traffic light features.
[0071] As described in the foregoing embodiments, the traffic light perception model is an end-to-end model. All output information from the traffic light perception module (including image features and traffic light features) is used as input information for the virtual light acquisition module, where it is processed to obtain the virtual light. Therefore, the traffic light perception model achieves a complete end-to-end process from environmental image input to virtual light output, with no information loss throughout the entire process of obtaining the virtual light from the environmental image, achieving lossless information transmission. Under lossless information transmission, even if errors occur in the traffic light features, it will not significantly affect the accuracy of the virtual light output by the traffic light perception model.
[0072] Furthermore, the image features from the bird's-eye view can reflect the characteristics of the intersection where the traffic lights are located (i.e., the environmental characteristics of the traffic lights). When the virtual light acquisition module acquires virtual lights based on the image features and traffic light features, it is equivalent to taking into account both the characteristics of the traffic lights themselves and the environmental characteristics, which further improves the accuracy of the virtual lights.
[0073] In some embodiments of this application, when obtaining a virtual light based on image features and traffic light features, the virtual light can be obtained in the following way: based on the model parameters of the traffic light perception model, obtain the query features of the traffic direction in the attention mechanism, where the traffic direction is the traffic direction in the virtual light; based on the query features, perform cross-attention processing on the traffic direction and image features to obtain a first processing result; perform cross-attention processing on the first processing result and traffic light features to obtain a second processing result, where the second processing result includes the indication state of the traffic direction; and obtain the virtual light based on the traffic direction and its indication state.
[0074] The Query feature is a model parameter of the traffic light perception model. Training the traffic light perception model can be understood as optimizing each model parameter to obtain its optimal value. The specific feature value of the Query feature is determined through model training, and the virtual light acquisition module can directly call the Query feature.
[0075] When performing cross-attention processing on traffic direction indication and image features, the image features are treated as key and value features of the attention mechanism, and attention processing is then performed based on the query features, key features, and value features to obtain the first processing result.
[0076] When performing cross-attention processing on the first processing result and the traffic light features, the first processing result is used as the Query feature in the attention mechanism, and the traffic light features are used as the Key feature and Value feature in the attention mechanism, respectively. Attention processing is performed based on the currently determined Query feature, Key feature, and Value feature to obtain the second processing result.
[0077] It should be noted that after determining the sources of the Query features, Key features, and Value features, conventional cross-attention processing methods can be used to perform cross-attention processing on traffic direction indication and image features, as well as cross-attention processing on the first processing result and traffic light features. This embodiment does not impose specific limitations on this.
[0078] Please refer to the appendix for further details. Figure 2 The virtual light acquisition module receives image features and traffic light features output by the traffic light sensing module. Using the query features of traffic direction indication, it processes the image features and traffic light features to obtain the indication status of the traffic direction in the virtual light. The indication status includes color, flashing status, countdown timer, and occlusion status. The occlusion status indicates whether the current traffic direction is obstructed. Furthermore, the virtual light acquisition module can also output the presence status of the traffic light. If the traffic light exists, the indication status of each traffic direction in the virtual light will be output simultaneously; if the traffic light does not exist, the indication status of each traffic direction in the virtual light will not be output.
[0079] The following is in conjunction with the appendix Figure 3 and attached Figure 4 This paper compares and explains the present application with existing traffic light sensing methods (i.e., traffic light sensing methods that combine sensing models with post-processing).
[0080] First, please refer to the appendix. Figure 3 , Figure 3 The main flow of an existing traffic light sensing method is illustrated by example. Figure 3As shown, existing traffic light perception methods include a perception stage and a post-processing stage. In the perception stage, a detection model is used to detect the light frame and light head in the image, and a classification model is used to detect the attributes of the light head. In the post-processing stage, the light frame and light head obtained by the detection model in multiple consecutive image frames are acquired. By associating and tracking the light frames, the same light frame in different images is linked together. Similarly, by associating and tracking the light heads, the same light head in different images is linked together and assigned a tracking ID. For each light head indicated by a tracking ID, each attribute of the light head in multiple consecutive image frames is smoothed to obtain accurate results for each attribute. Finally, the indication state of each traffic direction is determined based on the attributes of all light heads. When determining the indication state, candidate light frames located on or near the road in front of the vehicle are selected based on rules such as distance and angle. Finally, according to the correspondence rules between the light head shape and the traffic direction, the traffic direction corresponding to the light head shape of the candidate light frame is obtained. The state of the traffic direction (such as color, countdown numbers, and flashing status) is determined based on the attributes of the candidate light frames.
[0081] The existing traffic light sensing methods have the following drawbacks: (1) They are too sensitive to the detection results of the detection model and have a low tolerance for errors. If the detection results fluctuate, the post-processing stage may not be able to obtain the state of the traffic lights. (2) The post-processing stage only processes the features of the light frame and the light head itself and lacks the perception of the intersection where the traffic lights are located. Therefore, there are often cases of missed reports, false reports, and misreports. (3) Since it is impossible to cover all traffic light scenarios through rules, the generalization ability of the post-processing is poor and the upper limit is low. (4) Although the detection model can be iteratively upgraded through data-driven methods, since the post-processing adopts a rule-based method, it is difficult to iteratively upgrade the entire method through data-driven methods to improve the perception ability of traffic lights.
[0082] See appendix Figure 4 , Figure 4 The main flow of the traffic light sensing method provided in this application is illustrated by way of example. Figure 4 As shown, the method provided in this application includes only one perception stage. In the perception stage, a traffic light detection model is used to detect traffic lights in the environmental image, obtaining image features of the environmental image and traffic light features of the traffic lights in the environmental image. The image features and traffic light features are then processed to obtain virtual lights. After obtaining the virtual lights, they are output to the interaction module of a smart device for display, allowing users to intuitively view the indication status of each traffic direction in the virtual lights through the interaction module.
[0083] The method provided in this application has the following technical effects: (1) The traffic light perception model is an end-to-end model. There is no information loss in the entire process of obtaining the virtual light from the environmental image, and the lossless transmission of information is realized. Under the condition of lossless information transmission, even if there is an error in the traffic light features, it will not significantly affect the accuracy of the virtual light output by the traffic light perception model. (2) Image features can reflect the characteristics of the intersection where the traffic light is located (i.e., the environmental features of the traffic light). When the virtual light acquisition module obtains the virtual light based on the image features and the traffic light features, it is equivalent to taking into account both the characteristics of the traffic light itself and the environmental features at the same time, which further improves the accuracy of the virtual light. (3) Using the samples of the environmental image and their annotation information to iteratively train the traffic light perception model can continuously improve the perception ability of the model and realize the iterative upgrade of the entire traffic light perception method, that is, realize the iterative upgrade of the traffic light perception method through data-driven means.
[0084] The following describes an embodiment of the control method for the intelligent device provided in this application, which is executed by the intelligent device.
[0085] See appendix Figure 5 , Figure 5 This is a schematic flowchart illustrating the main steps of a control method for a smart device according to an embodiment of this application. Figure 5 As shown, the control method of the smart device in this application embodiment mainly includes the following steps S201 to S204.
[0086] Step S201: Obtain the virtual light corresponding to the traffic light in the environment where the smart device is located. Specifically, the traffic light sensing method described in the aforementioned method embodiments can be used to obtain the virtual light.
[0087] Step S202: Obtain the driving direction of the smart device.
[0088] The driving direction can be left turn, right turn, straight ahead, or U-turn. Specifically, the navigation path of the smart device can be obtained, and the driving direction of the smart device can be determined based on the navigation path. When enabling the autonomous driving mode or autonomous driving function of the smart device, it is necessary to set the starting point and destination of the smart device's journey. The navigation path can be understood as the driving path of the smart device from the starting point to the destination. This driving path is a road-level driving path planned based on the starting point, destination, and map information. For example, based on the navigation path and the smart device's location information, it is determined that the smart device will turn left at the intersection with the traffic light, that is, the driving direction is left turn.
[0089] Step S203: Based on the driving direction, query the indication status of the target direction from the virtual lights. The target direction is the traffic indication direction used to indicate the driving direction.
[0090] Step S204: Perform driving control on the smart device according to the indicated status.
[0091] A road includes at least one lane. Even if a road has multiple lanes, smart devices travel in one lane. Therefore, it is necessary to plan the travel path of smart devices in the lanes of the road, that is, the lane-level travel path, which can also be described as a planned trajectory.
[0092] After obtaining the driving direction indication status from the virtual lights, the planned trajectory of the smart device can be determined using this indication status, and then the smart device can be controlled to drive according to the planned trajectory. The planned trajectory includes multiple trajectory points within a certain period of time after the current moment.
[0093] For example, if a smart device is traveling straight at an intersection, and the virtual traffic light shows that the straight-ahead direction is red (indicating that passage is prohibited), then the smart device can be controlled to stop before the stop line at the intersection. Once the straight-ahead direction turns green (indicating that passage is permitted), the smart device can then proceed straight through the intersection. This driving decision will be used to plan the smart device's travel path (i.e., the planned trajectory) and control the smart device to travel according to the planned trajectory.
[0094] Based on the method described in steps S201 to S204 above, the virtual light can be used to accurately query the indication status of each traffic direction at the traffic light intersection, thereby enabling safe and reliable driving control of the intelligent device according to the indication status of the traffic direction, ensuring the safety of the intelligent device driving at the traffic light intersection.
[0095] It should be noted that although the steps in the above embodiments are described in a specific order, those skilled in the art will understand that in order to achieve the effect of this application, different steps do not necessarily have to be executed in such an order. They can be executed simultaneously (in parallel) or in other orders. These adjusted solutions are equivalent to the technical solutions described in this application and therefore will also fall within the protection scope of this application.
[0096] Those skilled in the art will understand that all or part of the processes in the method of the above-described embodiment can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable storage medium can include any entity or device capable of carrying the computer program code, a medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory, a random access memory, an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.
[0097] Another aspect of this application provides a computer-readable storage medium.
[0098] In one embodiment of a computer-readable storage medium according to this application, the computer-readable storage medium may be configured to store a program that performs the traffic light sensing method or the intelligent device control method of the above-described method embodiments. This program may be loaded and run by a processor to implement the traffic light sensing method or the intelligent device control method. For ease of explanation, only the parts related to the embodiments of this application are shown; for specific technical details not disclosed, please refer to the method section of the embodiments of this application. The computer-readable storage medium may be a storage device comprising various electronic devices. Optionally, in the embodiments of this application, the computer-readable storage medium is a non-transitory computer-readable storage medium.
[0099] Another aspect of this application provides a smart device.
[0100] In one embodiment of a smart device according to this application, the smart device may include at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program, which, when executed by the at least one processor, implements the methods described in any of the above embodiments. See Appendix Figure 6 , Figure 6 The image exemplarily illustrates a communication connection between memory 11 and processor 12 via a bus.
[0101] In some embodiments of this application, the smart device may further include at least one sensor for sensing information. The sensor is communicatively connected to any type of processor mentioned in this application. Optionally, the smart device may further include an autonomous driving system for guiding the smart device to drive autonomously or assisting in driving. The processor communicates with the sensor and / or the autonomous driving system to perform the methods described in any of the above embodiments.
[0102] The technical solution of this application has been described above with reference to one embodiment shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of this application is obviously not limited to these specific embodiments. Without departing from the principles of this application, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of this application.
Claims
1. A method for sensing traffic signals, characterized in that, The method includes: Acquire environmental images collected by the smart device, wherein the environmental images are images of the environment in which the smart device is located, and the environmental images include traffic lights; The environmental image is input into the traffic light perception model for processing to obtain the virtual light corresponding to the environmental image; in, The virtual lights are used for driving control of the smart device. The virtual lights include multiple traffic direction indicators and the indication status of each traffic direction indicator. The traffic direction indicators are used to indicate the driving direction of the smart device. The traffic light perception model is trained based on samples of the environmental image and their annotation information, the annotation information including the virtual light corresponding to the sample.
2. The method according to claim 1, characterized in that, The traffic light perception model includes a traffic light perception module and a virtual light acquisition module; The traffic light sensing module is configured to: perform traffic light sensing on the environmental image to obtain the traffic light features of the traffic lights in the environmental image; The virtual light acquisition module is configured to acquire the virtual light corresponding to the environmental image based on the traffic light characteristics.
3. The method according to claim 2, characterized in that, The traffic light features include the illuminated light frame and the attributes of the illuminated light frame, wherein there is at least one illuminated lamp head within the illuminated light frame, and the attributes include the type, color, and countdown digits of the lamp head.
4. The method according to claim 2, characterized in that, The traffic light sensing module includes a feature extraction network and a traffic light detection network; The feature extraction network is configured to: extract features from the environmental image to obtain the image features of the environmental image from a bird's-eye view; The traffic light detection network is configured to detect traffic lights on image features from the bird's-eye view to obtain the traffic light features.
5. The method according to claim 4, characterized in that, The virtual light acquisition module is configured to: The virtual light is obtained based on the image features and the traffic light features.
6. The method according to claim 5, characterized in that, The step of obtaining the virtual light based on the image features and the traffic light features includes: Based on the model parameters of the traffic light perception model, the query features of traffic direction in the attention mechanism are obtained, wherein the traffic direction is the traffic direction in the virtual light; Based on the query features, cross-attention processing is performed on the traffic direction indication and the image features to obtain a first processing result; The first processing result is cross-attention processing is performed on the traffic light features to obtain a second processing result, which includes the indication status of the traffic direction. The virtual light is obtained based on the traffic direction and its status.
7. The method according to claim 1 or 6, characterized in that, The traffic direction indication status includes color, flashing status, and countdown numbers.
8. A control method for an intelligent device, characterized in that, The method includes: Using the traffic light sensing method described in any one of claims 1 to 7, the virtual light corresponding to the traffic light in the environment where the smart device is located is obtained; Obtain the driving direction of the intelligent device; Based on the driving direction, query the indication status of the target direction from the virtual lights, where the target direction is the traffic indication direction used to indicate the driving direction; The intelligent device is controlled for driving based on the indicated state.
9. A smart device, characterized in that, include: At least one processor; And, a memory communicatively connected to the at least one processor; The memory stores a computer program, which, when executed by the at least one processor, implements the traffic light sensing method according to any one of claims 1 to 7, or the intelligent device control method according to claim 8.
10. A computer-readable storage medium storing a plurality of program codes, characterized in that, The program code is adapted to be loaded and run by a processor to perform the traffic light sensing method according to any one of claims 1 to 7, or the intelligent device control method according to claim 8.