An image processing method, device and intelligent automobile

Abstract

This application discloses an image processing method applicable to intelligent vehicles and intelligent connected vehicles. The method includes: acquiring an image to be processed; inputting the image to be processed into a first neural network to obtain a first prediction result; when the first prediction result indicates that a first region of the image to be processed is a lane line, obtaining a region of interest (ROI) for a detected object in the image to be processed based on height information and the first region; the height information includes a pre-set physical height of the detected object; and the ROI is used by a second neural network to obtain candidate bounding boxes and classifications of the detected object. The solution provided in this application can improve the accuracy of object recognition at intersections, such as improving the accuracy of traffic light recognition.

Description

Technical Field

[0001] This application relates to the field of image processing, and more particularly to an image processing method, apparatus, and intelligent vehicle. Background Technology

[0002] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to have perception, reasoning, and decision-making capabilities. Research in the field of AI includes robotics, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, and fundamental AI theories.

[0003] Autonomous driving is a mainstream application of artificial intelligence. Autonomous driving technology relies on the collaborative efforts of computer vision, radar, monitoring devices, and global positioning systems to enable motor vehicles to operate autonomously without human intervention. Autonomous vehicles use various computing systems to help transport passengers from one location to another. Some autonomous vehicles may require initial or continuous input from an operator (such as a navigator, driver, or passenger). Autonomous vehicles allow operators to switch from manual mode to autonomous mode or a mode in between. Because autonomous driving technology eliminates the need for human drivers, it theoretically can effectively avoid human driving errors, reduce traffic accidents, and improve highway transportation efficiency. Therefore, autonomous driving technology is receiving increasing attention.

[0004] As a crucial component of traffic operations, improving the accuracy of traffic light detection is of great significance for autonomous driving. Summary of the Invention

[0005] This application provides an image processing method, apparatus, and intelligent vehicle to improve the accuracy of object recognition at intersections, such as improving the accuracy of traffic light recognition.

[0006] To solve the above-mentioned technical problems, this application provides the following technical solution:

[0007] This application provides an image processing method applicable to the field of autonomous driving within the field of artificial intelligence. The method may include: acquiring an image to be processed; inputting the image to be processed into a first neural network to obtain a first prediction result. The first neural network may be a neural network used to perform image segmentation tasks. Neural networks used for image segmentation tasks in related technologies can be employed in this application, such as the first neural network including but not limited to: special convolutional neural networks (SCNN), fully convolutional networks (FCN), U-Net, mask region convolutional neural networks (Mask-RCNN), and semantic segmentation networks (SegNet). The first prediction result indicates the probability that each pixel in the image to be processed belongs to a lane line, specifically the probability that each pixel belongs to a stop lane line and the probability that each pixel belongs to a guide lane line. The set of pixels whose probability of belonging to a stop lane line exceeds a preset threshold can be used to obtain the region of a stop lane line in the image to be processed. The set of pixels whose probability of belonging to a guide lane line exceeds a preset threshold can be used to obtain the region of a guide lane line in the image to be processed. When the first prediction result indicates that the first region of the image to be processed is a lane line, the region of interest (ROI) for the object to be detected in the image to be processed is obtained based on the height information and the first region. The height information may include a pre-set physical height of the object to be detected. The ROI is used by a second neural network to obtain candidate boxes and classifications of the object to be detected. The second neural network can be a neural network used for object recognition tasks, including but not limited to convolutional neural networks (CNNs), deep neural networks (DNNs), YOLO v3 (version number, representing the third version), and single-shot multibox detectors (SSDs). In this application, the ROI refers to the region to be processed (also referred to as the matted region) delineated from the image to be processed using a bounding box. This ROI is input into the second neural network to output candidate boxes and classifications of the object to be detected. Determining the ROI includes determining the location of interest, the length of the ROI, and the width of the ROI.This application proposes a method for obtaining regions of interest (ROIs) using lane lines. Specifically, the location and length of the ROI can be determined based on the lane lines, and the width of interest can be determined based on the height of the object to be detected. This method utilizes lane lines to select regions corresponding to intersection segments in the image to be processed, which helps improve the accuracy of object detection at intersections.

[0008] In one possible implementation, the lane lines in the first region may include stop lines. Obtaining the region of interest (ROI) of the image to be processed based on height information and the first region may include: obtaining the length of the stop line in the image to be processed; obtaining the length of the ROI based on the length of the stop line in the image to be processed; obtaining the length of the object to be detected in the image to be processed based on height information and a scale bar, where the scale bar indicates the proportional relationship between the length of the object to be detected in the image to be processed and the physical height of the object; and obtaining the width of the ROI based on the length of the object to be detected in the image to be processed. When the image to be processed includes stop lane lines, it indicates that the vehicle is at an intersection or is about to enter an intersection. Obtaining the ROI based on the stop lane lines can effectively select the region corresponding to the intersection in the image to be processed, which is beneficial to improving the detection accuracy of objects at intersections.

[0009] In one possible implementation, the first region may include multiple first pixels, and the probability that each of the multiple first pixels belongs to a stop line exceeds a first preset threshold. The stop line is composed of multiple first pixels. Obtaining the length of the stop line in the image to be processed may include: obtaining the length of the stop line in the image to be processed based on the distance between the two farthest pixels among the multiple first pixels. This implementation provides a specific method for obtaining the length of the stop line in the image to be processed, increasing the diversity of solutions.

[0010] In one possible implementation, the method may further include: acquiring a first distance, which is the distance between the object to be detected and the vehicle; acquiring a second distance, which is the distance between the stop line and the lower edge of the image to be processed; and acquiring a scale bar based on the first and second distances. This implementation provides a specific method for acquiring the scale bar, increasing the versatility of the approach.

[0011] In one possible implementation, the lane lines in the first region may further include at least two guide lane lines, and the method may further include: obtaining the widths of any two adjacent guide lane lines in the image to be processed; and obtaining a scale bar based on the widths of the two adjacent guide lane lines in the image to be processed and a preset physical width of the two guide lane lines. This implementation provides another specific method for obtaining the scale bar, increasing the diversity of solutions.

[0012] In one possible implementation, obtaining the length of the region of interest (ROI) based on the length of the stop line in the image to be processed may include: obtaining the length of the ROI based on the distance between a first intersection point and a second intersection point, where the first intersection point is the intersection of a first guide lane line and one end of the stop line in the image to be processed, and the second intersection point is the intersection of a second guide lane line and the other end of the stop line in the image to be processed, and the first and second guide lane lines are the two guide lane lines that are furthest apart among at least two guide lane lines. This implementation provides a specific method for obtaining the length of the ROI based on the length of the stop line, increasing the versatility of the solutions.

[0013] In one possible implementation, the position of the lower edge of the region of interest is determined based on the position of the stop line in the image to be processed. This implementation simplifies the calculation process by directly determining the position of the lower edge of the region of interest based on the position of the stop line in the image to be processed.

[0014] In one possible implementation, the lane lines in the first region may include at least two guide lane lines but may not include stop lines. Obtaining the region of interest (ROI) of the image to be processed based on height information and the first region may include: obtaining the length of the ROI based on the distance between a third intersection point and a fourth intersection point, where the third intersection point is the intersection of a first guide lane line and one end of a first line segment in the image to be processed, the second intersection point is the intersection of a second guide lane line and the other end of the first line segment in the image to be processed, the first and second guide lane lines are the two guide lane lines furthest apart among the at least two guide lane lines, the first line segment is a line segment passing through a second pixel, and the second pixel is the pixel corresponding to the highest point of the shortest guide lane line among the at least two guide lane lines in the image to be processed. The length of the object to be detected in the image to be processed is obtained based on height information and a scale bar, where the scale bar indicates the proportional relationship between the length of the object to be detected in the image to be processed and the physical height of the object. The width of the ROI is obtained based on the length of the object to be detected in the image to be processed. In this implementation, the acquired image to be processed does not include stop lane lines but does include guide lane lines. Therefore, the position of the lower edge of the Region of Interest (ROI) and the length of the ROI region can be determined based on the positional relationship between the guide lane lines in the image to be processed. This ensures that even when no stop lane lines are detected, a suitable ROI region can be determined based on the guide lane lines, allowing the selection of the region corresponding to the intersection segment in the image to be processed, thus improving the accuracy of object detection at intersection segments.

[0015] In one possible implementation, the first line segment is parallel to the lower edge of the image to be processed.

[0016] In one possible implementation, the lane lines in the first region may include at least two guide lane lines but may not include stop lines. The position of the lower edge of the region of interest is determined based on the position of a first line segment in the image to be processed. The first line segment occupies a preset length of pixels, and one end of the first line segment intersects with a first guide lane line, while the other end intersects with a second guide lane line. The first and second guide lane lines are the two guide lane lines furthest apart among the at least two guide lane lines. In this implementation, for cases where the acquired image to be processed does not include stop lane lines but includes guide lane lines, another method for obtaining the region of interest based on the guide lane lines is provided.

[0017] In one possible implementation, obtaining the region of interest (ROI) of the image to be processed based on height information and a first region may include: obtaining the length of the ROI based on the length of a first line segment; obtaining the length of the object to be detected in the image to be processed based on the height information and a scale bar, wherein the scale bar indicates the proportional relationship between the length of the object to be detected in the image to be processed and the physical height of the object to be detected; and obtaining the width of the ROI based on the length of the object to be detected in the image to be processed.

[0018] In one possible implementation, if the resolution of the region of interest (ROI) obtained based on the height information and the first region is greater than a second preset threshold, the method may further include: compressing the resolution of the ROI to the second preset threshold. In this possible implementation, the size of the ROI may be too large. To reduce the computational load on the second neural network, the ROI may be compressed, and the compressed ROI is then input into the second neural network.

[0019] In one possible implementation, if the resolution of the region of interest (ROI) obtained based on the height information and the first region is less than a second preset threshold, the method may further include: performing super-resolution processing on the ROI to increase the resolution of the ROI to the second preset threshold. In this possible implementation, super-resolution processing can also be performed on the ROI to improve the image quality of the ROI, and the super-resolution processed ROI can be input into a second neural network to improve the second neural network's object detection performance.

[0020] In one possible implementation, the object to be detected may include a traffic light.

[0021] A second aspect of this application provides an image processing apparatus, which may include: an acquisition module for acquiring an image to be processed; an image segmentation module for inputting the image to be processed into a first neural network to obtain a first prediction result; and a region of interest module, further configured to, when the first prediction result indicates that a first region of the image to be processed is a lane line, acquire a region of interest for a detected object in the image to be processed based on height information and the first region, wherein the height information may include a preset physical height of the detected object, and the region of interest is used by a second neural network to acquire candidate bounding boxes and classify the detected object.

[0022] In one possible implementation, the lane lines in the first region may include stop lines. A region of interest (ROI) module is specifically configured to: acquire the length of the stop lines in the image to be processed; acquire the length of the ROI based on the length of the stop lines in the image to be processed; acquire the length of the object to be detected in the image to be processed based on height information and a scale bar, the scale bar indicating the proportional relationship between the length of the object to be detected in the image to be processed and the physical height of the object; and acquire the width of the ROI based on the length of the object to be detected in the image to be processed.

[0023] In one possible implementation, the first region may include a plurality of first pixels, and the probability that each of the plurality of first pixels belongs to a stop line exceeds a first preset threshold. The stop line is composed of a plurality of first pixels. The region of interest module is specifically used to: obtain the length of the stop line in the image to be processed based on the distance between the two farthest pixels among the plurality of first pixels.

[0024] In one possible implementation, the region of interest module is further configured to: acquire a first distance, which is the distance between the object to be detected and the vehicle; acquire a second distance, which is the distance between the stop line and the lower edge of the image to be processed; and acquire a scale bar based on the first and second distances.

[0025] In one possible implementation, the lane lines in the first region may further include at least two guide lane lines. The region of interest module is further configured to: obtain the width of any two adjacent guide lane lines in the image to be processed; and obtain a scale bar based on the width of any two adjacent guide lane lines in the image to be processed and a preset physical width of the two guide lane lines.

[0026] In one possible implementation, the region of interest module is specifically used to: obtain the length of the region of interest based on the distance between a first intersection point and a second intersection point, wherein the first intersection point is the intersection point of a first guide lane line and one end of a stop line in the image to be processed, and the second intersection point is the intersection point of a second guide lane line and the other end of a stop line in the image to be processed, and the first guide lane line and the second guide lane line are the two guide lane lines that are furthest apart among at least two guide lane lines.

[0027] In one possible implementation, the position of the lower edge of the region of interest is determined based on the position of the stop line in the image to be processed.

[0028] In one possible implementation, the lane lines in the first region may include at least two guide lane lines but may not include stop lines. The region of interest (ROI) module is specifically configured to: obtain the length of the ROI based on the distance between a third intersection point and a fourth intersection point, where the third intersection point is the intersection of a first guide lane line and one end of a first line segment in the image to be processed, the second intersection point is the intersection of a second guide lane line and the other end of the first line segment in the image to be processed, the first and second guide lane lines are the two guide lane lines furthest apart among the at least two guide lane lines, the first line segment is a line segment passing through a second pixel, and the second pixel is the pixel corresponding to the highest point of the shortest guide lane line among the at least two guide lane lines in the image to be processed; obtain the length of the object to be detected in the image to be processed based on height information and a scale bar, where the scale bar indicates the proportional relationship between the length of the object to be detected in the image to be processed and the physical height of the object to be detected; and obtain the width of the ROI based on the length of the object to be detected in the image to be processed.

[0029] In one possible implementation, the first line segment is parallel to the lower edge of the image to be processed.

[0030] In one possible implementation, the lane lines in the first region may include at least two guide lane lines but may not include stop lines. The position of the lower edge of the region of interest is determined based on the position of the first line segment in the image to be processed. The first line segment occupies a preset length of pixels, and one end of the first line segment intersects with the first guide lane line, while the other end of the first line segment intersects with the second guide lane line. The first guide lane line and the second guide lane line are the two guide lane lines that are furthest apart among the at least two guide lane lines.

[0031] In one possible implementation, the region of interest (ROI) module is specifically configured to: obtain the length of the ROI based on the length of the first line segment; obtain the length of the object to be detected in the image to be processed based on height information and a scale bar, wherein the scale bar indicates the proportional relationship between the length of the object to be detected in the image to be processed and the physical height of the object; and obtain the width of the ROI based on the length of the object to be detected in the image to be processed.

[0032] In one possible implementation, a compression module may also be included, which is used to compress the resolution of the region of interest to the second preset threshold if the resolution of the region of interest obtained based on the height information and the first region is greater than the second preset threshold.

[0033] In one possible implementation, a super-resolution processing module may also be included. The super-resolution processing module is used to perform super-resolution processing on the region of interest if the resolution of the region of interest obtained based on the height information and the first region is less than a second preset threshold, so as to increase the resolution of the region of interest to the second preset threshold.

[0034] In one possible implementation, the object to be detected may include a traffic light.

[0035] A third aspect of this application provides an image processing apparatus that may include a processor and a memory coupled together, the memory storing program instructions that, when executed by the processor, implement the method described in the first aspect.

[0036] A fourth aspect of this application provides a computer-readable storage medium that may include a program, which, when run on a computer, causes the computer to perform the methods described in the first aspect.

[0037] The fifth aspect of this application provides a computer program product that, when run on a computer, enables the computer to perform the methods described in the first aspect.

[0038] The sixth aspect of this application provides a chip coupled to a memory for executing a program stored in the memory to perform the method described in the first aspect.

[0039] The seventh aspect of this application provides an intelligent vehicle that may include processing circuitry and storage circuitry configured to perform the methods described in the first aspect.

[0040] The solution provided in this application targets images acquired from vehicles. If the acquired image includes lane lines, the region of interest (ROI) for the detected object in the image is obtained based on the lane lines. Specifically, the position and length of the ROI can be determined based on the lane lines, and the width of interest (ROI) can be determined based on the height of the detected object. When the image includes stop lane lines, it indicates that the vehicle is at an intersection or about to enter an intersection. The ROI is obtained based on the stop lane lines. For example, if the lane lines include stop lane lines, the position of the lower edge of the ROI and the length of the ROI are determined based on the position of the stop lane lines in the image. This effectively selects the region corresponding to the intersection in the image, improving the accuracy of object detection at intersections. If the acquired image does not include stop lane lines but includes guide lane lines, the position of the lower edge of the ROI and the length of the ROI can be determined based on the positional relationship between the guide lane lines in the image. Even when no stop lane line is detected, a suitable ROI area can be determined based on the guide lane line, and the corresponding area of ​​the intersection segment can be selected in the image to be processed, thereby improving the detection accuracy of the object to be detected at the intersection segment.

[0041] Furthermore, after obtaining the position and size of the region of interest (ROI) in the image to be processed, the ROI can be used as a matting region in the image to be processed. This matting region is then input into a second neural network, which determines the candidate bounding boxes and classification of the object to be detected based on the matting region. In some possible implementations, super-resolution processing can be performed on the matting region to improve its image quality. The super-resolution processed matting region is then input into the second neural network to improve its object detection performance. In some possible implementations, the size of the matting region may be too large. To reduce the computational load on the second neural network, the matting region can be compressed before being input into the second neural network. Attached Figure Description

[0042] Figure 1 A schematic diagram of the structure of an autonomous vehicle provided in an embodiment of this application;

[0043] Figure 2 A schematic flowchart of an image processing method provided in an embodiment of this application;

[0044] Figure 3 A flowchart illustrating another image processing method provided in an embodiment of this application;

[0045] Figure 4-a This is a schematic diagram of one method for obtaining the stop line length in an embodiment of this application;

[0046] Figure 4-b This is a schematic diagram of another method for obtaining the stop line length in an embodiment of this application;

[0047] Figure 4-c This is a schematic diagram of one method for obtaining the scale in an embodiment of this application;

[0048] Figure 4-d This is a schematic diagram of another method for obtaining the scale in the embodiments of this application;

[0049] Figure 5 A flowchart illustrating another image processing method provided in an embodiment of this application;

[0050] Figure 6 This is a schematic diagram illustrating an application scenario of an image processing method provided in this application;

[0051] Figure 7 A flowchart illustrating another image processing method provided in an embodiment of this application;

[0052] Figure 8 A schematic diagram illustrating another application scenario of the image processing method provided in this application;

[0053] Figure 9 A flowchart illustrating another image processing method provided in an embodiment of this application;

[0054] Figure 10 A schematic diagram illustrating another application scenario of the image processing method provided in this application;

[0055] Figure 11 A schematic flowchart of an image processing method provided in an embodiment of this application;

[0056] Figure 12 This is a schematic diagram for selecting a region of interest in an image to be processed.

[0057] Figure 13 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of this application;

[0058] Figure 14 This is another schematic diagram of the image processing apparatus provided in the embodiments of this application;

[0059] Figure 15 This is a schematic diagram of another structure of an autonomous vehicle provided in an embodiment of this application;

[0060] Figure 16 This is a schematic diagram of a chip structure provided in an embodiment of this application. Detailed Implementation

[0061] This application provides an image processing method that obtains the region of interest (ROI) of an image to be processed based on lane lines. The solution provided by this application can effectively improve the accuracy of object recognition in intersection scenes.

[0062] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0063] To facilitate understanding of this solution, the embodiments of this application first combine... Figure 1 For an introduction to the structure of autonomous vehicles, please refer to [link / reference needed]. Figure 1 , Figure 1This is a schematic diagram of the structure of an autonomous vehicle provided in an embodiment of this application. The autonomous vehicle 100 is configured in a fully or partially autonomous driving mode. For example, the autonomous vehicle 100 can control itself while in autonomous driving mode, and can determine the current state of the vehicle and its surrounding environment through human operation, determine the possible behavior of at least one other vehicle in the surrounding environment, and determine the confidence level corresponding to the probability of the other vehicle performing the possible behavior, and control the autonomous vehicle 100 based on the determined information. When the autonomous vehicle 100 is in autonomous driving mode, it can also be set to operate without human interaction.

[0064] The autonomous vehicle 100 may include various subsystems, such as a mobility system 102, a sensor system 104, a control system 106, one or more peripheral devices 108, a power supply 110, a computer system 112, and a user interface 116. Optionally, the autonomous vehicle 100 may include more or fewer subsystems, and each subsystem may include multiple components. Furthermore, each subsystem and component of the autonomous vehicle 100 may be interconnected via wired or wireless means.

[0065] The mobility system 102 may include components that provide powered motion for the autonomous vehicle 100. In one embodiment, the mobility system 102 may include an engine 118, an energy source 119, a transmission 120, and wheels 121.

[0066] Engine 118 can be an internal combustion engine, an electric motor, an air-compressed engine, or other combinations of engines, such as a hybrid engine consisting of a gasoline engine and an electric motor, or a hybrid engine consisting of an internal combustion engine and an air-compressed engine. Engine 118 converts energy source 119 into mechanical energy. Examples of energy source 119 include gasoline, diesel, other petroleum-based fuels, propane, other compressed gas-based fuels, ethanol, solar panels, batteries, and other sources of electricity. Energy source 119 can also provide energy to other systems of the autonomous vehicle 100. Transmission 120 transmits mechanical power from engine 118 to wheels 121. Transmission 120 may include a gearbox, a differential, and a drive shaft. In one embodiment, transmission 120 may also include other components, such as a clutch. The drive shaft may include one or more axles that can be coupled to one or more wheels 121.

[0067] Sensor system 104 may include several sensors for sensing information about the environment surrounding autonomous vehicle 100. For example, sensor system 104 may include a global positioning system 122 (the positioning system may be GPS, BeiDou, or other positioning systems), an inertial measurement unit (IMU) 124, radar 126, a laser rangefinder 128, and a camera 130. Sensor system 104 may also include sensors monitoring the internal systems of autonomous vehicle 100 (e.g., in-vehicle air quality monitor, fuel gauge, oil temperature gauge, etc.). Sensing data from one or more of these sensors can be used to detect objects and their corresponding characteristics (position, shape, orientation, speed, etc.). This detection and identification is a key function for the safe operation of autonomous vehicle 100.

[0068] The positioning system 122 can be used to estimate the geographical location of the autonomous vehicle 100. An IMU 124 is used to sense changes in the position and orientation of the autonomous vehicle 100 based on inertial acceleration. In one embodiment, the IMU 124 can be a combination of an accelerometer and a gyroscope. A radar 126 can use radio signals to sense objects in the surrounding environment of the autonomous vehicle 100, specifically millimeter-wave radar or lidar. In some embodiments, in addition to sensing objects, the radar 126 can also be used to sense the speed and / or direction of travel of objects. A laser rangefinder 128 can use lasers to sense objects in the environment in which the autonomous vehicle 100 is located. In some embodiments, the laser rangefinder 128 may include one or more laser sources, a laser scanner, and one or more detectors, as well as other system components. A camera 130 can be used to capture multiple images of the surrounding environment of the autonomous vehicle 100. The camera 130 can be a still camera or a video camera.

[0069] The control system 106 controls the operation of the autonomous vehicle 100 and its components. The control system 106 may include various components, including a steering system 132, a throttle 134, a braking unit 136, a computer vision system 140, a path control system 142, and an obstacle avoidance system 144.

[0070] The steering system 132 is operable to adjust the forward direction of the autonomous vehicle 100. For example, in one embodiment, it may be a steering wheel system. The throttle 134 controls the operating speed of the engine 118 and thus the speed of the autonomous vehicle 100. The braking unit 136 controls the deceleration of the autonomous vehicle 100. The braking unit 136 may use friction to slow down the wheels 121. In other embodiments, the braking unit 136 may convert the kinetic energy of the wheels 121 into electrical current. The braking unit 136 may also take other forms to slow down the rotational speed of the wheels 121 to control the speed of the autonomous vehicle 100. The computer vision system 140 is operable to process and analyze images captured by the camera 130 to identify objects and / or features in the environment surrounding the autonomous vehicle 100. The objects and / or features may include traffic signals, road boundaries, and obstacles. The computer vision system 140 may use object recognition algorithms, Structure from Motion (SFM) algorithms, video tracking, and other computer vision techniques. In some embodiments, the computer vision system 140 may be used to map the environment, track objects, estimate the speed of objects, etc. The route control system 142 is used to determine the driving route and speed of the autonomous vehicle 100. In some embodiments, the route control system 142 may include a lateral planning module 1421 and a longitudinal planning module 1422, which are respectively used to combine data from the obstacle avoidance system 144, GPS 122, and one or more predetermined maps to determine the driving route and speed for the autonomous vehicle 100. The obstacle avoidance system 144 is used to identify, evaluate, and avoid or otherwise traverse obstacles in the environment of the autonomous vehicle 100, which may specifically be physical obstacles and virtual moving bodies that may collide with the autonomous vehicle 100. In one example, the control system 106 may add or alternatively include components other than those shown and described. Alternatively, some of the components shown above may be reduced.

[0071] The autonomous vehicle 100 interacts with external sensors, other vehicles, other computer systems, or users via peripheral devices 108. Peripheral devices 108 may include a wireless communication system 146, an onboard computer 148, a microphone 150, and / or a speaker 152. In some embodiments, peripheral devices 108 provide a means for the user of the autonomous vehicle 100 to interact with a user interface 116. For example, the onboard computer 148 may provide information to the user of the autonomous vehicle 100. The user interface 116 may also operate the onboard computer 148 to receive user input. The onboard computer 148 may be operated via a touchscreen. In other cases, peripheral devices 108 may provide a means for the autonomous vehicle 100 to communicate with other devices located within the vehicle. For example, the microphone 150 may receive audio (e.g., voice commands or other audio input) from the user of the autonomous vehicle 100. Similarly, the speaker 152 may output audio to the user of the autonomous vehicle 100. The wireless communication system 146 may communicate wirelessly with one or more devices, either directly or via a communication network. For example, the wireless communication system 146 may use 3G cellular communication, such as code division multiple access (CDMA), EVDO, Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), or 4G cellular communication, such as long term evolution (LTE), or 5G cellular communication. The wireless communication system 146 may utilize a wireless local area network (WLAN) for communication. In some embodiments, the wireless communication system 146 may utilize an infrared link, Bluetooth, or ZigBee to communicate directly with devices. Other wireless protocols, such as various vehicle communication systems, may also be used. For example, the wireless communication system 146 may include one or more dedicated short range communications (DSRC) devices that may enable public and / or private data communication between vehicles and / or roadside stations.

[0072] Power source 110 can provide power to various components of autonomous vehicle 100. In one embodiment, power source 110 can be a rechargeable lithium-ion or lead-acid battery. One or more such battery packs can be configured to provide power to various components of autonomous vehicle 100. In some embodiments, power source 110 and energy source 119 can be implemented together, as is the case in some fully electric vehicles.

[0073] Some or all of the functions of the autonomous vehicle 100 are controlled by a computer system 112. The computer system 112 may include at least one processor 113, which executes instructions 115 stored in a non-transitory computer-readable medium such as memory 114. The computer system 112 may also be multiple computing devices controlling individual components or subsystems of the autonomous vehicle 100 in a distributed manner. The processor 113 may be any conventional processor, such as a commercially available central processing unit (CPU). Alternatively, the processor 113 may be a dedicated device such as an application-specific integrated circuit (ASIC) or other hardware-based processor. Although... Figure 1The processor, memory, and other components of the computer system 112 in the same block are functionally illustrated; however, those skilled in the art will understand that the processor or memory may actually include multiple processors or memories not stored in the same physical housing. For example, memory 114 may be a hard disk drive or other storage media located in a housing different from that of computer system 112. Therefore, references to processor 113 or memory 114 will be understood to include a collection of processors or memories that may or may not operate in parallel. Unlike using a single processor to perform the steps described herein, some components, such as steering and deceleration components, may each have their own processor that performs calculations only related to the component's specific function. In various aspects described herein, processor 113 may be located remotely from and wirelessly communicate with the autonomous vehicle 100. In other aspects, some of the processes described herein are executed on processor 113 located within the autonomous vehicle 100, while others are executed by remote processor 113, including taking the necessary steps to perform a single operation. In some embodiments, memory 114 may contain instructions 115 (e.g., program logic) that can be executed by processor 113 to perform various functions of the autonomous vehicle 100, including those described above. Memory 114 may also contain additional instructions, including instructions to send data to, receive data from, interact with, and / or control one or more of the driving system 102, sensor system 104, control system 106, and peripheral devices 108. For example, to change lanes to the right, a human driver would need to perform the following steps: First, consider safety and traffic regulations to determine the timing of the lane change; second, plan a driving trajectory; third, control the accelerator, brake, and steering wheel to allow the vehicle to travel along the predetermined trajectory. These operations correspond to an autonomous vehicle and can be executed by the autonomous vehicle's behavior planner (BP), motion planner (MoP), and control, respectively. The BP is responsible for issuing high-level decisions, the MoP is responsible for planning the expected trajectory and speed, and the control is responsible for operating the accelerator, brake, and steering wheel to allow the autonomous vehicle to follow the target trajectory and reach the target speed. It should be understood that the related operations performed by the behavior planner, motion planner, and motion controller can be as follows: Figure 1 The processor 113 shown executes instructions 115 in memory 114, which can be used to instruct the circuit control system 142. In this application embodiment, the behavior planner, motion planner, and motion controller are sometimes collectively referred to as the planning and control module.

[0074] In addition to instructions 115, memory 114 may also store data such as road maps, route information, vehicle position, direction, speed, and other such vehicle data, as well as other information. This information may be used by the autonomous vehicle 100 and computer system 112 during operation of the autonomous vehicle 100 in autonomous, semi-autonomous, and / or manual modes. User interface 116 is used to provide information to or receive information from a user of the autonomous vehicle 100. Optionally, user interface 116 may include one or more input / output devices within a set of peripheral devices 108, such as wireless communication system 146, onboard computer 148, microphone 150, and speaker 152.

[0075] Computer system 112 can control the functions of autonomous vehicle 100 based on input received from various subsystems (e.g., driving system 102, sensor system 104, and control system 106) and from user interface 116. For example, computer system 112 can utilize input from control system 106 to control steering system 132 to avoid obstacles detected by sensor system 104 and obstacle avoidance system 144. In some embodiments, computer system 112 is operable to provide control over many aspects of autonomous vehicle 100 and its subsystems.

[0076] Alternatively, one or more of these components may be installed separately from or associated with the autonomous vehicle 100. For example, the memory 114 may exist partially or completely separately from the autonomous vehicle 100. The components may be communicatively coupled together in a wired and / or wireless manner.

[0077] Optionally, the components described above are merely examples. In actual applications, components in each of the above modules may be added or removed as needed. Figure 1 This should not be construed as a limitation on the embodiments of this application. An autonomous vehicle traveling on a road, such as the autonomous vehicle 100 described above, can identify objects in its surrounding environment to determine adjustments to its current speed. These objects can be other vehicles, traffic control equipment, or other types of objects. In some examples, each identified object can be considered independently, and based on the object's individual characteristics, such as its current speed, acceleration, and distance from the vehicle, the speed adjustment to be made by the autonomous vehicle can be determined.

[0078] Optionally, the autonomous vehicle 100 or the computing device associated with the autonomous vehicle 100, such as Figure 1The computer system 112, computer vision system 140, and memory 114 can predict the behavior of the identified objects based on the characteristics of the identified objects and the state of the surrounding environment (e.g., traffic, rain, ice on the road, etc.). Optionally, each identified object depends on the behavior of each other, so all identified objects can be considered together to predict the behavior of a single identified object. The autonomous vehicle 100 can adjust its speed based on the predicted behavior of the identified objects. In other words, the autonomous vehicle 100 can determine what steady state the vehicle will need to adjust to (e.g., accelerate, decelerate, or stop) based on the predicted behavior of the objects. In this process, other factors can also be considered in determining the speed of the autonomous vehicle 100, such as the lateral position of the autonomous vehicle 100 on the road it is traveling on, the curvature of the road, the proximity of static and dynamic objects, etc. In addition to providing instructions to adjust the speed of the autonomous vehicle, the computing device can also provide instructions to modify the steering angle of the autonomous vehicle 100 so that the autonomous vehicle 100 follows a given trajectory and / or maintains a safe lateral and longitudinal distance from objects near the autonomous vehicle 100 (e.g., cars in adjacent lanes on the road).

[0079] The aforementioned autonomous vehicle 100 can be a car, truck, motorcycle, bus, ship, airplane, helicopter, lawnmower, recreational vehicle, amusement park vehicle, construction equipment, tram, golf cart, and train, etc., and this application embodiment does not impose any special limitations.

[0080] Based on the above description, embodiments of this application provide an image processing method that can be applied to... Figure 1 Among the 100 autonomous vehicles shown in the figure.

[0081] See Figure 2 This is a flowchart illustrating an image processing method provided in an embodiment of this application.

[0082] like Figure 2 As shown, an image processing method provided in this application may include the following steps:

[0083] 201. Obtain the image to be processed.

[0084] The vehicle can acquire images to be processed through sensor system 104. For example, the vehicle can acquire images to be processed through camera 130. The images to be processed are used to represent the environment around the vehicle. In one possible implementation, the vehicle can acquire environmental information about its surroundings in real time through camera 130, that is, acquire images to be processed in real time. In one possible implementation, when the vehicle detects that it is about to enter an intersection, it begins to acquire environmental information about its surroundings through camera 130, that is, it begins to acquire images to be processed when it detects that it is about to enter an intersection.

[0085] In one possible implementation, the acquired images to be processed can be filtered to obtain images with a signal-to-noise ratio (SNR) that meet preset conditions. Different filtering methods can be used to delete data that does not meet the SNR requirement, thus obtaining data that does. Duplicate images to be processed can also be deleted.

[0086] 202. Input the image to be processed into the first neural network to obtain the first prediction result.

[0087] The first neural network can be a neural network used to perform image segmentation tasks. Neural networks used for image segmentation tasks in related technologies can all be used in the embodiments of this application. For example, the first neural network includes, but is not limited to: special convolutional neural networks (SCNN), fully convolutional networks (FCN), U-Net, mask region convolutional neural networks (Mask-RCNN), and semantic segmentation networks (SegNet). The first prediction result indicates the probability that each pixel in the image to be processed belongs to a lane line, specifically the probability that each pixel belongs to a stop lane line and the probability that each pixel belongs to a guide lane line. The set of pixels whose probability of belonging to a stop lane line exceeds a preset threshold can be used to obtain the region of a stop lane line in the image to be processed. The set of pixels whose probability of belonging to a guide lane line exceeds a preset threshold can be used to obtain the region of a guide lane line in the image to be processed.

[0088] Whether lane lines are included in the image to be processed can be determined by a first neural network. If lane lines are included, they are segmented from the image. For example, the following is a lane line detection method: The image to be processed is input into the first neural network for feature extraction. The extracted features (each feature map is pre-divided into multiple grids) are then decoded by a prediction head model to generate dense line clusters (i.e., multiple predicted lane lines). Finally, the line clusters are sorted according to the confidence score of each predicted lane line (also known as the confidence score of the grid, which reflects whether a lane line crosses the grid and what the probability of crossing the grid is. Grids with a confidence score greater than a preset value are used to predict lane lines, while grids with a confidence score lower than a preset value are considered to have no contribution to the prediction). The predicted lane line with the highest confidence score is used as the baseline, and the lane lines are grouped together if the distance between other predicted lane lines and the baseline is less than a threshold. The line clusters are divided into several groups in a similar manner, and the baseline of each group is taken as the final detection result of a real lane line for that group. It should be noted that those skilled in the art can choose the lane line detection method according to the actual situation, and the embodiments of this application do not limit this.

[0089] 203. When the first prediction result indicates that the first region of the image to be processed is a lane line, the region of interest of the object to be detected in the image to be processed is obtained based on the height information and the first region.

[0090] Height information includes the preset physical height of the object to be detected. For example, if the object to be detected is a traffic light, the height information includes the preset actual height of the traffic light. For instance, if the height of a traffic light is typically 6-7 meters, the object height of the traffic light can be preset to 7 meters. Similarly, if the object to be detected is a car, the height information includes the preset actual height of the car. For instance, if the height of a car is typically 1.4 to 1.6 meters, the object height of the car can be preset to 1.6 meters.

[0091] The region of interest (ROI) is used by the second neural network to obtain candidate bounding boxes and classifications of the object to be detected. This second neural network can be a neural network used for object recognition tasks, including but not limited to convolutional neural networks (CNNs), deep neural networks (DNNs), YOLO v3 (version number, representing the third version), and single-shot multibox detectors (SSDs). In this application, the ROI refers to the region to be processed (also referred to as the matted region) delineated as a bounding box from the image to be processed, and this ROI is input into the second neural network to output candidate bounding boxes and classifications of the object to be detected. Determining the ROI includes determining the location of interest, the length of the ROI, and the width of the ROI.

[0092] The solution provided in this application targets images acquired from vehicles. If the acquired image includes lane lines, the region of interest (ROI) for the detected object in the image is obtained based on the lane lines. Specifically, the position and length of the ROI can be determined based on the lane lines, and the width of interest (ROI) can be determined based on the height of the detected object. When the image includes stop lane lines, it indicates that the vehicle is at an intersection or about to enter an intersection. The ROI is obtained based on the stop lane lines. For example, if the lane lines include stop lane lines, the position of the lower edge of the ROI and the length of the ROI are determined based on the position of the stop lane lines in the image. This effectively selects the region corresponding to the intersection in the image, improving the accuracy of object detection at intersections. If the acquired image does not include stop lane lines but includes guide lane lines, the position of the lower edge of the ROI and the length of the ROI can be determined based on the positional relationship between the guide lane lines in the image. This method ensures that even when no stop lane line is detected, a suitable ROI region can be determined based on the guide lane lines, and the corresponding area of ​​the intersection segment can be selected in the image to be processed, thereby improving the detection accuracy of the target object in the intersection segment. It should be noted that this application sometimes refers to stop lane lines as stop lines, and the two terms have the same meaning.

[0093] In one possible implementation, the object to be detected is a traffic light, such as a traffic light. The solution provided in this application can effectively improve the accuracy of traffic light detection. One factor limiting the accuracy of traffic light detection is that, for images acquired at the same focal length, the number of pixels occupied by a traffic light in the image is much smaller than the number of pixels occupied by other objects (such as people or vehicles) in the image. Before being input into the neural network, the image needs to be compressed to reduce its size, thereby reducing the amount of data required for the neural network to process the image. Since the proportion of pixels occupied by a traffic light in the image is already small, compression may further reduce the number of pixels occupied by the traffic light, greatly increasing the difficulty of traffic light detection. To ensure that the number of pixels occupied by a traffic light in the image is not compressed due to compression and to ensure that the amount of data required for the neural network to process the image is reduced, one approach is to select the Region of Interest (ROI) of the traffic light and define the ROI region of the traffic light. The input domain is fed into the neural network, enabling the network to detect traffic lights based on the Region of Interest (ROI). Since the ROI of traffic lights is constantly changing as a vehicle approaches, for example, as the vehicle moves closer to a traffic light, the ROI moves upwards. Therefore, selecting the correct ROI is crucial for improving the accuracy of traffic light detection. Current technologies generally use fixed ROI regions, which cannot adapt to the constantly changing nature of traffic light ROIs. Furthermore, current methods typically obtain the ROI of traffic lights using high-precision maps and the vehicle's GPS location information. However, this method is vulnerable when GPS positioning is inaccurate, or when GPS signals or high-precision maps are unavailable. In such cases, the ROI cannot be obtained, potentially leading to the failure to detect traffic lights and increasing safety hazards. The solution provided in this application can obtain the ROI region of traffic lights through lane lines, without being limited by GPS signals and high-precision maps. Furthermore, it obtains the ROI region of traffic lights based on information about the lane lines in the image to be processed, such as the position information of the stop lane lines, the length of the stop lane lines, and the positional relationship between the guide lane lines. This makes the ROI region of traffic lights dynamically changing. Combined with the actual physical height of the traffic lights, it can better select the ROI region of traffic lights in the image to be processed. The ROI region obtained through the solution provided in this application is input into a neural network, enabling the neural network to perform traffic light detection based on the ROI region obtained through the solution provided in this application, which can effectively improve the accuracy of traffic light detection.

[0094] Figure 2The corresponding embodiments describe that lane lines may include stop lane lines and guide lane lines. Depending on whether the lane lines include stop lane lines, there may be different implementation methods for obtaining the region of interest of the object to be detected in the image to be processed based on height information and the first region. The following describes how to obtain the region of interest of the object to be detected in the image to be processed based on lane lines, using several typical implementation methods.

[0095] 1. The image to be processed includes stop lane lines.

[0096] See Figure 3 This is a flowchart illustrating another image processing method provided in an embodiment of this application.

[0097] like Figure 3 As shown, another image processing method provided in this application may include the following steps:

[0098] 301. Obtain the image to be processed.

[0099] 302. Input the image to be processed into the first neural network to obtain the first prediction result.

[0100] Steps 301 and 302 can be referred to Figure 2 Steps 201 and 202 in the corresponding embodiments will be understood and will not be repeated here.

[0101] 303. When the first prediction result indicates that the first region of the image to be processed is a lane line, and the lane line includes a stop line, the length of the stop line in the image to be processed is obtained.

[0102] The first prediction result indicates the probability that each pixel in the image to be processed belongs to the stop lane line. The region occupied by pixels whose probability of belonging to the stop line exceeds a preset threshold is called Region 1. Region 1 in the first region can then be used to represent the position of the stop lane line in the image to be processed. See also Figure 4-a , Figure 4-a This is a schematic diagram of one scheme for obtaining the length of the stop line in an embodiment of this application. In the image to be processed, the stop line is composed of multiple first pixels, where each first pixel is a pixel included in a first region. In one possible implementation, the length of the stop line in the image to be processed can be obtained based on the distance between the two farthest pixels among the first pixels.

[0103] In one possible implementation, the first prediction result indicates that region 1 of the image to be processed is a lane line, which also includes guide lane lines. See below. Figure 4-b , Figure 4-bThis is a schematic diagram of another scheme for obtaining the stop line length in an embodiment of this application. In this method, multiple pixels can be selected from region 1, and a straight line fitting is performed based on these multiple pixels to obtain the fitted straight line segment. Furthermore, the first prediction result indicates the probability that each pixel belongs to the guide lane line. Each guide lane line has its own probability map. Taking a guide lane line as an example, the probability map of that guide lane line indicates the probability that each pixel belongs to that guide lane line. The region occupied by pixels whose probability of belonging to the guide lane line exceeds a preset threshold is called region 2. Region 2 in the first region can then be used to represent the position of the guide lane line in the image to be processed. Multiple pixels can be selected from region 2, and a straight line fitting is performed based on these multiple pixels. For each guide lane line, the fitted line segment can be obtained in the above manner to obtain multiple fitted line segments. In some possible implementations, curve fitting can also be performed on multiple pixels. The length of the stop line in the image to be processed is obtained based on the distance between the first and second intersection points. The first intersection point is the intersection of one end of the curved line segment corresponding to the first guide lane line and one end of the straight line segment corresponding to the stop line in the image to be processed. The second intersection point is the intersection of the other end of the curved line segment corresponding to the second guide lane line and one end of the straight line segment corresponding to the stop line in the image to be processed. The first and second guide lane lines are the two guide lane lines that are furthest apart from each other among at least two guide lane lines. Redetermining the length of the stop lane line using the guide lane lines helps to obtain a more accurate length of the stop lane line and reduce errors.

[0104] 304. Obtain the length of the region of interest based on the length of the stop line in the image to be processed.

[0105] In one possible implementation, the length of the stop line is the length of the region of interest. In another possible implementation, the length of the stop line can be modified, for example, by adding a preset pixel distance to the stop line length to obtain the length of the region of interest.

[0106] 305. Obtain the length of the object to be detected in the image to be processed based on the height information and scale.

[0107] The scale bar is used to indicate the proportional relationship between the length of the object to be detected in the image to be processed and the physical height of the object.

[0108] The method provided in this application can obtain the scale in a variety of possible ways, including but not limited to the following two methods:

[0109] In one possible implementation, a first distance is obtained, which is the distance between the object to be detected and the vehicle. This distance can be obtained in various ways, such as using radar 126. Related technologies have various methods for obtaining this distance, and all embodiments of this application can employ these methods, including but not limited to monocular ranging and binocular ranging. A second distance is obtained, which is the distance between the stop line and the lower edge of the image to be processed, for example, by referring to... Figure 4-c The distance between the stop line and the bottom edge of the image to be processed indicates the distance between the object to be detected (e.g., a traffic light) and the vehicle. A scale is obtained based on the first and second distances. The scale allows us to determine the actual physical length corresponding to one pixel, and thus the pixel length occupied by the physical height of the object to be detected in the image to be processed. The physical height of the object to be detected is referenced to... Figure 2 The physical height of the object to be detected described in step 203 of the corresponding embodiment is understood and will not be repeated here.

[0110] In one possible implementation, when the first prediction result indicates that a first region of the image to be processed is a lane line, and that the lane line also includes guide lane lines, for example, region 2 of the first region indicates that it includes at least two guide lane lines. See also Figure 4-d The process involves obtaining the widths of any two adjacent guide lane lines in the image to be processed. The widths of any two adjacent guide lane lines in the image to be processed are used to obtain a scale based on the widths of these two adjacent guide lane lines and a preset physical width of the two guide lane lines. In this implementation, the widths of any two adjacent guide lane lines in the image to be processed indicate the width of the lane in the image. Combined with the actual lane width (e.g., a lane width typically 3.5 meters, which can be preset), the ratio of these two values ​​yields the actual physical length corresponding to the length of one pixel. Furthermore, the pixel length occupied by the physical height of the object to be detected in the image to be processed can be obtained.

[0111] 305. Obtain the width of the region of interest based on the length of the object to be detected in the image to be processed.

[0112] In one possible implementation, the length of the object to be detected in the image to be processed is the width of the region of interest. In another possible implementation, the length of the object to be detected in the image to be processed can be modified, for example, by increasing the length of the object to be detected in the image to be processed by a preset pixel distance to obtain the width of the region of interest.

[0113] In one possible implementation, the position of the lower edge of the region of interest is determined based on the position of the stop line in the image to be processed. For example, the lower edge of the region of interest is the line segment between the first intersection point and the second intersection point of the stop line obtained in step 303.

[0114] Through the above steps, the length and width of the region of interest (ROI) can be obtained, and thus the size of the ROI can be determined. For intersections, stop lines and traffic lights often appear together in the image to be processed. As vehicles move, the areas of the stop lines and traffic lights in the image constantly change. The solution provided in this application obtains the position of the ROI in the image by the position of the stop lines, and obtains the size of the ROI by the length of the stop lines. Since the area of ​​the stop lines in the image constantly changes, the position and size of the obtained ROI also change accordingly. This allows the ROI to accurately encompass the intersection scene, which is beneficial for identifying objects at intersections, such as traffic lights, and improves the accuracy of object recognition at intersections.

[0115] II. The image to be processed does not include stop lane lines.

[0116] In some scenarios, the image to be processed may not include the stop lane line. In this case, the region of interest can be obtained based on the guide lane line. Different methods for obtaining the region of interest can be used depending on the shape of the guide lane line. The following describes several typical implementation methods.

[0117] See Figure 5 This is a flowchart illustrating another image processing method provided in an embodiment of this application.

[0118] like Figure 5 As shown, another image processing method provided in this application may include the following steps:

[0119] 501. Obtain the image to be processed.

[0120] 502. Input the image to be processed into the first neural network to obtain the first prediction result.

[0121] Steps 501 and 502 can be referred to Figure 2 Steps 201 and 202 in the corresponding embodiments will be understood and will not be repeated here.

[0122] 503. The first prediction result indicates that the lane lines in the first region of the image to be processed include at least two guide lane lines but do not include stop lines. The length of the region of interest is obtained based on the distance between the third and fourth intersection points.

[0123] The first prediction result indicates the probability that each pixel belongs to a guide lane line. Each guide lane line has its own probability map. Taking a guide lane line as an example, the probability map of that guide lane line indicates the probability that each pixel belongs to that guide lane line. The area occupied by pixels whose probability of belonging to that guide lane line exceeds a preset threshold is called region 2. Region 2 in the first region can then be used to represent the position of that guide lane line in the image to be processed (the pixels included in region 2 form a guide lane line). In one possible implementation, multiple pixels can be selected from region 2, and curve fitting can be performed based on these multiple pixels. For each guide lane line, the fitted curve segment can be obtained in the above manner, and the fitted curve segment can be considered as a guide lane line. The third intersection point is the intersection of one end of the first guide lane line and the first straight line segment in the image to be processed. The second intersection point is the intersection of the other end of the second guide lane line and the first straight line segment in the image to be processed. The first guide lane line and the second guide lane line are the two guide lane lines that are farthest apart among at least two guide lane lines. The first straight line segment is a straight line segment that passes through the second pixel. The second pixel is the pixel corresponding to the highest point of the shortest guide lane line among at least two guide lane lines in the image to be processed. The following is combined with... Figure 6 For further explanation, please refer to the following: Figure 6 In some possible scenarios, the image to be processed includes at least two guide lane lines, at least one of which is missing, resulting in inconsistent lengths between the two lane lines. The missing lane line may be due to an actual absence of the lane line or it may be due to processing by an image segmentation neural network; this application does not limit the specific circumstances of the missing lane line. When it is determined that at least one lane line in the image to be processed is missing, the length of the region of interest is obtained based on the pixel corresponding to the highest point of the shortest lane line in the image, the straight line segment passing through the pixel corresponding to the highest point, and the intersection points (the third and fourth intersection points) of the leftmost and rightmost lane lines. In one possible implementation, the line segment between the third and fourth intersection points is the lower edge of the region of interest.

[0124] 504. Obtain the length of the object to be detected in the image to be processed based on the height information and scale.

[0125] Step 504 can be referred to Figure 3 The steps 305 in the corresponding embodiments will be understood and will not be repeated here.

[0126] 505. Obtain the width of the region of interest based on the length of the object to be detected in the image to be processed.

[0127] In one possible implementation, the length of the object to be detected in the image to be processed is the width of the region of interest. In another possible implementation, the length of the object to be detected in the image to be processed can be modified, for example, by increasing the length of the object to be detected in the image to be processed by a preset pixel distance to obtain the width of the region of interest.

[0128] Depend on Figure 5 As can be seen from the corresponding embodiments, in some possible scenarios, the image to be processed does not include stop lane lines, making it impossible to select the area corresponding to the intersection segment in the image based on the stop lane lines. To ensure that the area corresponding to the intersection segment can be selected from the image even when the image to be processed does not include stop lane lines, and to guarantee that the obtained area is as complete as possible, the position of the lower edge of the region of interest is obtained based on the pixel corresponding to the highest point of the shortest guide lane line in the image to further determine the size of the region of interest.

[0129] exist Figure 5 In corresponding embodiments, at least two guide lane lines in the image to be processed may contain missing guide lane lines. In some possible implementations, at least two guide lane lines in the image to be processed may be complete guide lane lines without any missing lines. Furthermore, in some possible implementations, at least two of the at least two lane lines in the image to be processed may intersect. Wherein, if the difference between the x-coordinate of a pixel belonging to one guide lane line and the x-coordinate of a pixel belonging to another guide lane line is within a preset range, then the two guide lane lines can be considered to intersect. The following describes how to determine the size and location of the region of interest in these scenarios, using a specific implementation method.

[0130] See Figure 7 This is a flowchart illustrating another image processing method provided in an embodiment of this application.

[0131] like Figure 7 As shown, another image processing method provided in this application may include the following steps:

[0132] 701. Obtain the image to be processed.

[0133] 702. Input the image to be processed into the first neural network to obtain the first prediction result.

[0134] Steps 701 and 702 can be referred to Figure 2 Steps 201 and 202 in the corresponding embodiments will be understood and will not be repeated here.

[0135] 703. The lane lines in the first region include at least two guide lane lines but do not include stop lines. The position of the lower edge of the region of interest is determined based on the position of the first line segment in the image to be processed.

[0136] The first prediction result indicates the probability that each pixel belongs to a guide lane line. Each guide lane line has its own probability map. Taking a guide lane line as an example, the probability map of that guide lane line indicates the probability that each pixel belongs to that guide lane line. The area occupied by pixels whose probability of belonging to that guide lane line exceeds a preset threshold is called region 2. Region 2 in the first region can then be used to represent the position of that guide lane line in the image to be processed (the pixels included in region 2 form a guide lane line). In one possible implementation, multiple pixels can be selected from region 2, and curve fitting can be performed based on these multiple pixels. For each guide lane line, the fitted curve segment can be obtained in the above manner, and the fitted curve segment can be considered as a guide lane line.

[0137] The first line segment occupies a preset length of pixels, where the preset length can be a range of lengths, and any line segment within that range is the first line segment. Alternatively, the preset length can be a fixed length. One end of the first line segment intersects with a first guide lane line, and the other end intersects with a second guide lane line. The first and second guide lane lines are the two guide lane lines furthest apart among at least two guide lane lines. In one possible implementation, the first line segment occupies 300 pixels. Assuming the intersection point of the first line segment and the first guide lane line is intersection point 1, and the intersection point of the first line segment and the second guide lane line is intersection point 2, then the difference between the x-coordinate of intersection point 1 and the x-coordinate of intersection point 2 is the preset length of pixels. For example, the difference between the x-coordinate of intersection point 1 and the x-coordinate of intersection point 2 is 300 pixels. Another possible implementation is that the preset length is a range of lengths, referring to... Figure 8 To understand this, the leftmost and rightmost guide lane lines are the two guide lane lines that are furthest apart from each other among at least two guide lane lines. There may be countless line segments between the leftmost and rightmost guide lane lines, for example... Figure 8Line segments 1, 2, and 3 are considered. Suppose that the pixel length occupied by line segment 1 does not meet the condition (e.g., it's outside the preset length range, specifically exceeding the maximum length within the preset range); the pixel length occupied by line segment 3 does not meet the condition (e.g., it's outside the preset length range, specifically less than the minimum length within the preset length range); and the pixel length occupied by line segment 2 meets the condition (e.g., it's within the preset length range). Then, any line segment satisfying the preset length range is arbitrarily selected as the first line segment (e.g., line segment 2). The position of the lower edge of the region of interest (ROI) is determined based on the first line segment. For example, the distance between the lower edge of the ROI and the first line segment on the image to be processed should not exceed a preset threshold.

[0138] 704. Obtain the length of the region of interest based on the length of the first line segment.

[0139] In one possible implementation, the length of the first line segment is the length of the region of interest. In another possible implementation, the length of the first line segment is processed to obtain the length of the region of interest, for example, by increasing the length of the first line segment by a preset pixel distance to obtain the length of the region of interest.

[0140] 705. Obtain the length of the object to be detected in the image to be processed based on the height information and scale.

[0141] Step 705 can be referred to Figure 3 The steps 305 in the corresponding embodiments will be understood and will not be repeated here.

[0142] 706. Obtain the width of the region of interest based on the length of the object to be detected in the image to be processed.

[0143] In one possible implementation, the length of the object to be detected in the image to be processed is the width of the region of interest. In another possible implementation, the length of the object to be detected in the image to be processed can be modified, for example, by increasing the length of the object to be detected in the image to be processed by a preset pixel distance to obtain the width of the region of interest.

[0144] Depend on Figure 7As can be seen from the corresponding embodiments, in some possible scenarios, the image to be processed does not include stop lane lines, making it impossible to select the area corresponding to the intersection segment in the image based on the stop lane lines. To ensure that the image to be processed can still select the possible area corresponding to the intersection segment from the image even when the stop lane lines are not included, and to ensure that the obtained area corresponding to the complete intersection segment is obtained while also preventing it from being too small, thus resulting in a small region of interest, the lower edge position of the region of interest is obtained based on the intersection point of a line segment corresponding to a preset pixel length with the leftmost and rightmost guide lane lines, further determining the size of the region of interest.

[0145] The above describes how to obtain the location and size of the region of interest (ROI) in the image to be processed. After obtaining the location and size of the ROI in the image to be processed, the ROI can be used as a matting region in the image to be processed. The matting region is then input into a second neural network, which determines the candidate bounding box and classification of the object to be detected based on the matting region. In some possible implementations, super-resolution processing can be performed on the matting region to improve the image quality of the matting region. The super-resolution processed matting region is then input into the second neural network to improve the second neural network's object detection performance. In some possible implementations, the size of the matting region may be too large. To reduce the computational load on the second neural network, the matting region can be compressed, and the compressed matting region is then input into the second neural network. A specific embodiment will be described below.

[0146] See Figure 9 This is a flowchart illustrating another image processing method provided in an embodiment of this application.

[0147] like Figure 9 As shown, another image processing method provided in this application may include the following steps:

[0148] 901. Obtain the image to be processed.

[0149] 902. Input the image to be processed into the first neural network to obtain the first prediction result.

[0150] Steps 901 and 902 can be referred to Figure 2 Steps 201 and 202 in the corresponding embodiments will be understood and will not be repeated here.

[0151] 903. When the first prediction result indicates that the first region of the image to be processed is a lane line, the region of interest of the object to be detected in the image to be processed is obtained based on the height information and the first region.

[0152] Figure 2 , Figure 3 , Figure 5 , Figure 7 The corresponding embodiment describes a method for obtaining the region of interest (ROI) of the object to be detected in the image to be processed based on height information and a first region. Figure 9 All corresponding implementation examples can be used, and will not be repeated here.

[0153] 904. If the resolution of the region of interest obtained based on the height information and the first region is greater than the second preset threshold, the resolution of the region of interest is compressed to the second preset threshold.

[0154] For example, if the preset resolution is 896*512 pixels, and the resolution of the region of interest (ROI) obtained from the height information and the first region is greater than 896*512 pixels, then the obtained ROI is compressed to reduce its resolution to 896*512 pixels. The preset resolution is related to the input of the second neural network; for example, if the input format of the second neural network is 896*512 pixels, then the preset resolution is set to 896*512 pixels. Various methods can be used to compress the image to a specified resolution, and this application does not limit this approach. For example, averaging multiple adjacent pixels to obtain a single pixel can achieve the purpose of image compression.

[0155] 905. If the resolution of the region of interest obtained based on the height information and the first region is less than the second preset threshold, perform super-resolution processing on the region of interest to increase the resolution of the region of interest to the second preset threshold.

[0156] For example, if the preset resolution is 896*512 pixels, and the resolution of the region of interest (ROI) obtained based on the height information and the first region is less than 896*512 pixels, then super-resolution processing is performed on the obtained ROI to increase its resolution to 896*512 pixels. The preset resolution is related to the input of the second neural network; for example, if the input format of the second neural network is 896*512 pixels, then the preset resolution is set to 896*512 pixels. There are various possible implementations for performing super-resolution processing on an image to improve its quality to a specified pixel level, and this application does not limit this approach. For example, super-resolution processing can be performed using deep learning networks such as super-resolution convolutional neural networks (SRCNN) and region-based fast convolutional neural networks (FSRCNN). Specifically, a bicubic interpolation algorithm can be used on the ROI to improve its resolution.

[0157] After super-resolution processing, the image details are more apparent, which is beneficial for improving the accuracy of object detection. (See also...) Figure 10 The second neural network processes the region of interest (ROI) to obtain candidate bounding boxes and categories of the objects to be detected within it. Based on the coordinates of any pixel within the ROI, the ROI can be re-merged into the image to be processed. For example, since each pixel in the image to be processed has corresponding coordinates, the ROI can be re-merged into the image to be processed based on the coordinates of the top-left pixel of the ROI. This allows the candidate bounding boxes and categories of the objects to be detected to be displayed in the image to be processed.

[0158] Reference Figure 11This is a flowchart illustrating an image processing method provided in an embodiment of this application. The method involves acquiring an image to be processed, determining whether a stop lane line exists in the image, and if so, acquiring a region of interest (ROI) based on the stop lane line and guide lane lines. Specifically, the lower edge (length and position) of the ROI is determined based on the line segment between the two intersection points of the stop lane line and the leftmost and rightmost lane lines, and the width of the ROI is obtained based on the scale and the actual physical height of the object to be detected. If the image to be processed does not contain stop lane lines, it is further determined whether the shortest guide lane line in the image intersects with other guide lane lines. If they intersect, the lower edge (length and position) of the ROI is obtained based on a target line segment. The target line segment is 300 pixels long, with one end intersecting the leftmost lane line and the other end intersecting the rightmost lane line. It should be noted that the 300-pixel length is merely illustrative; the length of the target line segment can be determined based on a threshold input to the second neural network. If the lines do not intersect, the lower edge (length and position) of the region of interest (ROI) is obtained based on the line segment between the two intersection points of the target line segment and the leftmost and rightmost lane lines. The width of the ROI is then determined based on the scale and the actual physical height of the object to be detected. The target line segment is parallel to the lower edge of the image being processed, and the pixel corresponding to the highest point of the shortest lane line in the image is also considered. After obtaining the ROI, the relationship between its resolution and a preset resolution is compared. If the ROI resolution is greater than the preset resolution, the ROI is compressed to bring its resolution down to the preset resolution. If the ROI resolution is less than the preset resolution, super-resolution processing is performed to increase the ROI resolution to the preset resolution.

[0159] See Figure 12 This is a schematic diagram for selecting the region of interest in the image to be processed. The lower edge (length and position) of the region of interest is determined by the line segment between the two intersection points of the stop lane line and the leftmost lane line and the rightmost lane line. The width of the region of interest is obtained by the scale and the actual physical height of the object to be detected.

[0160] In one possible implementation, the region of interest (ROI) can be displayed via an in-vehicle device or projected onto the windshield. The ROI always includes the area corresponding to the intersection segment. The ROI obtained according to the solution provided in this application will only include traffic lights that affect the driving state of the vehicle's lane. Therefore, detecting traffic lights within this ROI will only output one decision result.

[0161] The above describes an image processing method provided by the embodiments of this application. The image processing method provided by this application can effectively select the area corresponding to the intersection segment in the image to be processed, which is beneficial to improving the detection accuracy of the object to be detected at the intersection segment.

[0162] exist Figures 2 to 11 Based on the corresponding embodiments, in order to better implement the above-described solutions of this application, related equipment for implementing the above solutions is also provided below. See details. Figure 13 , Figure 13 This is a schematic diagram of an image processing apparatus provided in an embodiment of this application. The image processing apparatus may include an acquisition module 131, an image segmentation module 132, and a region of interest module 133.

[0163] In one possible implementation, the acquisition module 131 is used to acquire the image to be processed. The image segmentation module 132 is used to input the image to be processed into a first neural network to obtain a first prediction result. The region of interest module 133 is further used to acquire the region of interest of the object to be detected in the image to be processed based on the height information and the first region when the first prediction result indicates that the first region of the image to be processed is a lane line. The height information may include a preset physical height of the object to be detected. The region of interest is used by a second neural network to acquire candidate boxes and classify the object to be detected.

[0164] In one possible implementation, the lane lines in the first region may include stop lines. The region of interest module 133 is specifically configured to: acquire the length of the stop lines in the image to be processed; acquire the length of the region of interest based on the length of the stop lines in the image to be processed; acquire the length of the object to be detected in the image to be processed based on height information and a scale bar, the scale bar indicating the proportional relationship between the length of the object to be detected in the image to be processed and the physical height of the object to be detected; and acquire the width of the region of interest based on the length of the object to be detected in the image to be processed.

[0165] In one possible implementation, the first region may include a plurality of first pixels, and the probability that each of the plurality of first pixels belongs to a stop line exceeds a first preset threshold. The stop line is composed of a plurality of first pixels. The region of interest module 133 is specifically used to: obtain the length of the stop line in the image to be processed based on the distance between the two farthest pixels among the plurality of first pixels.

[0166] In one possible implementation, the region of interest module 133 is further configured to: acquire a first distance, which is the distance between the object to be detected and the vehicle; acquire a second distance, which is the distance between the stop line and the lower edge of the image to be processed; and acquire a scale bar based on the first and second distances.

[0167] In one possible implementation, the lane lines in the first region may further include at least two guide lane lines. The region of interest module 133 is further configured to: obtain the width of any two adjacent guide lane lines in the image to be processed; and obtain a scale bar based on the width of any two adjacent guide lane lines in the image to be processed and a preset physical width of the two guide lane lines.

[0168] In one possible implementation, the region of interest module 133 is specifically used to: obtain the length of the region of interest based on the distance between the first intersection point and the second intersection point, wherein the first intersection point is the intersection point of the first guide lane line and one end of the stop line in the image to be processed, and the second intersection point is the intersection point of the second guide lane line and the other end of the stop line in the image to be processed, and the first guide lane line and the second guide lane line are the two guide lane lines that are furthest apart among at least two guide lane lines.

[0169] In one possible implementation, the position of the lower edge of the region of interest is determined based on the position of the stop line in the image to be processed.

[0170] In one possible implementation, the lane lines in the first region may include at least two guide lane lines but may not include stop lines. The region of interest module 133 is specifically configured to: obtain the length of the region of interest based on the distance between a third intersection point and a fourth intersection point, where the third intersection point is the intersection of a first guide lane line and one end of a first line segment in the image to be processed, the second intersection point is the intersection of a second guide lane line and the other end of the first line segment in the image to be processed, the first and second guide lane lines are the two guide lane lines furthest apart among the at least two guide lane lines, the first line segment is a line segment passing through a second pixel, and the second pixel is the pixel corresponding to the highest point of the shortest guide lane line among the at least two guide lane lines in the image to be processed; obtain the length of the object to be detected in the image to be processed based on height information and a scale bar, where the scale bar indicates the proportional relationship between the length of the object to be detected in the image to be processed and the physical height of the object to be detected; and obtain the width of the region of interest based on the length of the object to be detected in the image to be processed.

[0171] In one possible implementation, the first line segment is parallel to the lower edge of the image to be processed.

[0172] In one possible implementation, the lane lines in the first region may include at least two guide lane lines but may not include stop lines. The position of the lower edge of the region of interest is determined based on the position of the first line segment in the image to be processed. The first line segment occupies a preset length of pixels, and one end of the first line segment intersects with the first guide lane line, while the other end of the first line segment intersects with the second guide lane line. The first guide lane line and the second guide lane line are the two guide lane lines that are furthest apart among the at least two guide lane lines.

[0173] In one possible implementation, the region of interest module 133 is specifically configured to: obtain the length of the region of interest based on the length of the first line segment; obtain the length of the object to be detected in the image to be processed based on height information and a scale bar, wherein the scale bar indicates the proportional relationship between the length of the object to be detected in the image to be processed and the physical height of the object to be detected; and obtain the width of the region of interest based on the length of the object to be detected in the image to be processed.

[0174] In one possible implementation, a compression module may also be included, which is used to compress the resolution of the region of interest to the second preset threshold if the resolution of the region of interest obtained based on the height information and the first region is greater than the second preset threshold.

[0175] In one possible implementation, a super-resolution processing module may also be included. The super-resolution processing module is used to perform super-resolution processing on the region of interest if the resolution of the region of interest obtained based on the height information and the first region is less than a second preset threshold, so as to increase the resolution of the region of interest to the second preset threshold.

[0176] In one possible implementation, the object to be detected may include a traffic light.

[0177] See Figure 14 This is another schematic diagram of the image processing apparatus provided in the embodiments of this application. It includes a processor 1402 and a memory 1403.

[0178] The processor 1402 includes, but is not limited to, one or more of a central processing unit (CPU), a network processor (NP), an application-specific integrated circuit (ASIC), or a programmable logic device (PLD). The PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof. The processor 1402 is responsible for the communication line 1404 and general processing, and can also provide various functions, including timing, peripheral interface, voltage regulation, power management, and other control functions.

[0179] Memory 1403 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. Memory 1403 may exist independently and be connected to processor 1402 via communication line 1404. Memory 1403 may also be integrated with processor 1402. If the memory 1403 and the processor 1402 are independent devices, the memory 1403 and the processor 1402 can be connected, for example, the memory 1403 and the processor 1402 can communicate through a communication line. The communication line 1404 can also communicate with the processor 1402 directly.

[0180] The communication line 1404 may include any number of interconnected buses and bridges, and the communication line 1404 may link together various circuits including one or more processors 1402 represented by processor 1402 and memory represented by memory 1403. The communication line 1404 may also link together various other circuits such as peripheral devices, voltage regulators and power management circuits, which are well known in the art and therefore will not be described further in this application.

[0181] In one possible implementation, the image processing transpose may include a processor coupled to a memory, the memory storing program instructions that are executed by the processor when the stored program instructions are executed. Figures 2 to 11 The method described.

[0182] This application also provides an autonomous driving vehicle, in conjunction with the above-described embodiments. Figure 1 For a description, please refer to Figure 15 , Figure 15 This is a schematic diagram of the structure of an autonomous vehicle provided in an embodiment of this application, wherein the autonomous vehicle 100 may be equipped with... Figure 14 The image processing apparatus described in the corresponding embodiment is used to implement Figures 2 to 11 The corresponding embodiments describe the functions of the autonomous vehicle. Since in some embodiments, the autonomous vehicle 100 may also include communication functions, the autonomous vehicle 100 includes, in addition to... Figure 1 The components shown may further include a receiver 1201 and a transmitter 1202, wherein the processor 113 may include an application processor 1131 and a communication processor 1132. In some embodiments of this application, the receiver 1201, transmitter 1202, processor 113, and memory 114 may be connected via a bus or other means.

[0183] Processor 113 controls the operation of the autonomous vehicle. In specific applications, the various components of the autonomous vehicle 100 are coupled together through a bus system, which may include a data bus, as well as a power bus, control bus, and status signal bus. However, for clarity, all buses are referred to as the bus system in the diagram.

[0184] Receiver 1201 can be used to receive input digital or character information, and to generate signal inputs related to the settings and function control of the autonomous vehicle. Transmitter 1202 can be used to output digital or character information through the first interface; transmitter 1202 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; transmitter 1202 may also include a display device such as a display screen.

[0185] In this embodiment of the application, the application processor 1131 is used to execute... Figures 2 to 11 The image processing method executed by the autonomous vehicle or image processing device in the corresponding embodiment.

[0186] It should be noted that the specific implementation of the image processing method executed by the application processor 1131 and the beneficial effects thereof can be found in [reference needed]. Figures 2 to 11 The descriptions in the corresponding method embodiments will not be repeated here.

[0187] This application also provides a computer-readable storage medium storing a program for planning vehicle routes, which, when the vehicle is driven on a computer, causes the computer to execute the aforementioned... Figures 2 to 11 The steps performed by the autonomous vehicle (or image processing device) in the method described in the illustrated embodiment.

[0188] This application also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned actions. Figures 2 to 11 The steps performed by the autonomous vehicle (or image processing device) in the method described in the illustrated embodiment.

[0189] This application embodiment also provides a circuit system, the circuit system including a processing circuit, the processing circuit being configured to perform the aforementioned... Figures 2 to 11 The steps performed by the autonomous vehicle (or image processing device) in the method described in the illustrated embodiment.

[0190] The image processing device or autonomous vehicle provided in this application embodiment can specifically be a chip. The chip includes a processing unit and a communication unit. The processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in a storage unit to cause the chip within the server to perform the aforementioned operations. Figures 2 to 9 The illustrated embodiment describes a method for planning vehicle routes. Optionally, the storage unit is a storage unit within the chip, such as a register or cache. Alternatively, the storage unit can be a storage unit located outside the chip within the wireless access device, such as a read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, such as random access memory (RAM).

[0191] For details, please refer to Figure 16 , Figure 16 This is a schematic diagram of a chip provided in an embodiment of this application. The chip can be represented as a neural network processor (NPU) 130. The NPU 130 is mounted as a coprocessor on the host CPU, and tasks are assigned by the host CPU. The core part of the NPU is the arithmetic circuit 1303, which is controlled by the controller 1304 to extract matrix data from the memory and perform multiplication operations.

[0192] In some implementations, the arithmetic circuit 1303 internally includes multiple processing engines (PEs). In some implementations, the arithmetic circuit 1303 is a two-dimensional pulsating array. The arithmetic circuit 1303 can also be a one-dimensional pulsating array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 1303 is a general-purpose matrix processor.

[0193] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory 1302 and caches it in each PE of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory 1301 and performs matrix operations with matrix B. The partial result or the final result of the obtained matrix is ​​stored in the accumulator 1308.

[0194] Unified memory 1306 is used to store input and output data. Weight data is directly transferred to weight memory 1302 via direct memory access controller (DMAC) 1305. Input data is also transferred to unified memory 1306 via DMAC.

[0195] The bus interface unit (BIU) 1310 is used for interaction between the AXI bus and the DMAC and instruction fetch buffer (IFB) 1309.

[0196] BIU1310 is used by instruction fetch memory 1309 to fetch instructions from external memory, and also by memory cell access controller 1305 to fetch the original data of input matrix A or weight matrix B from external memory.

[0197] The DMAC is mainly used to move input data from external memory DDR to unified memory 1306, or to weight data to weight memory 1302, or to input data to input memory 1301.

[0198] The vector computation unit 1307 includes multiple arithmetic processing units that further process the output of the computation circuit as needed, such as vector multiplication, vector addition, exponential operations, logarithmic operations, size comparisons, etc. It is mainly used for computation in non-convolutional / fully connected layers of neural networks, such as batch normalization, pixel-level summation, and upsampling of feature planes.

[0199] In some implementations, the vector computation unit 1307 can store the processed output vector in the unified memory 1306. For example, the vector computation unit 1307 can apply linear and / or nonlinear functions to the output of the computation circuit 1303, such as performing linear interpolation on feature planes extracted by convolutional layers, or accumulating a vector of values ​​to generate activation values. In some implementations, the vector computation unit 1307 generates normalized values, pixel-level summed values, or both. In some implementations, the processed output vector can be used as activation input to the computation circuit 1303, for example, for use in subsequent layers of the neural network.

[0200] The instruction fetch buffer 1309 connected to the controller 1304 is used to store the instructions used by the controller 1304.

[0201] Unified memory 1306, input memory 1301, weighted memory 1302, and instruction fetch memory 1309 are all on-chip memories. External memory is proprietary to this NPU hardware architecture.

[0202] The operations of each layer in the recurrent neural network can be performed by the arithmetic circuit 1303 or the vector calculation unit 1307.

[0203] The processor mentioned above can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits used to control the execution of a program in the first aspect of the method.

[0204] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0205] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CLUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0206] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0207] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

Claims

1. An image processing method, characterized in that, include: Obtain the image to be processed; The image to be processed is input into a first neural network to obtain a first prediction result; When the first prediction result indicates that the first region of the image to be processed is a lane line, the region of interest of the object to be detected in the image to be processed is obtained according to the height information and the first region. The height information includes the preset physical height of the object to be detected. The region of interest is used by the second neural network to obtain the candidate box and classification of the object to be detected. When the lane lines in the first region include stop lines, the step of obtaining the region of interest of the image to be processed based on the height information and the first region includes: Obtain the length of the stop line in the image to be processed; The length of the region of interest is obtained based on the length of the stop line in the image to be processed; The length of the object to be detected in the image to be processed is obtained based on the height information and the scale. The scale is used to indicate the proportional relationship between the length of the object to be detected in the image to be processed and the physical height of the object to be detected. The width of the region of interest is obtained based on the length of the object to be detected in the image to be processed.

2. The method according to claim 1, characterized in that, The first region includes a plurality of first pixels, wherein the probability of each of the plurality of first pixels belonging to the stop line exceeds a first preset threshold, the stop line is composed of the plurality of first pixels, and obtaining the length of the stop line in the image to be processed includes: The length of the stop line in the image to be processed is obtained based on the distance between the two farthest pixels among the plurality of first pixels.

3. The method according to claim 1, characterized in that, The method further includes: Obtain a first distance, which is the distance between the object to be detected and the vehicle; Obtain a second distance, which is the distance between the stop line and the lower edge of the image to be processed; The scale is obtained based on the first distance and the second distance.

4. The method according to claim 1, characterized in that, The lane markings in the first area also include at least two guide lane markings, and the method further includes: Obtain the width of any two adjacent guide lane lines in the image to be processed from the at least two guide lane lines; The scale is obtained based on the width of any two adjacent guide lane lines in the image to be processed and the preset physical width of the two guide lane lines.

5. The method according to claim 4, characterized in that, The step of obtaining the length of the region of interest based on the length of the stop line in the image to be processed includes: The length of the region of interest is obtained based on the distance between the first intersection point and the second intersection point. The first intersection point is the intersection point of the first guide lane line and one end of the stop line in the image to be processed. The second intersection point is the intersection point of the second guide lane line and the other end of the stop line in the image to be processed. The first guide lane line and the second guide lane line are the two guide lane lines that are farthest apart among the at least two guide lane lines.

6. The method according to any one of claims 1 to 5, characterized in that, The position of the lower edge of the region of interest is determined based on the position of the stop line in the image to be processed.

7. The method according to claim 1, characterized in that, When the lane lines in the first region include at least two guide lane lines but do not include stop lines, the step of obtaining the region of interest of the image to be processed based on the height information and the first region includes: The length of the region of interest is obtained based on the distance between the third and fourth intersection points. The third intersection point is the intersection of one end of the first guide lane line and the first line segment in the image to be processed. The fourth intersection point is the intersection of the other end of the second guide lane line and the first line segment in the image to be processed. The first guide lane line and the second guide lane line are the two guide lane lines that are farthest apart among the at least two guide lane lines. The first line segment is a line segment that passes through a second pixel. The second pixel is the pixel corresponding to the highest point of the shortest guide lane line among the at least two guide lane lines in the image to be processed. The length of the object to be detected in the image to be processed is obtained based on the height information and the scale. The scale is used to indicate the proportional relationship between the length of the object to be detected in the image to be processed and the physical height of the object to be detected. The width of the region of interest is obtained based on the length of the object to be detected in the image to be processed.

8. The method according to claim 7, characterized in that, The first line segment is parallel to the lower edge of the image to be processed.

9. The method according to claim 1, characterized in that, When the lane lines in the first region include at least two guide lane lines but do not include stop lines, the step of obtaining the region of interest of the image to be processed based on the height information and the first region includes: The position of the lower edge of the region of interest is determined based on the position of the first line segment in the image to be processed. The first line segment occupies a preset length of pixels, and one end of the first line segment intersects with a first guide lane line, and the other end of the first line segment intersects with a second guide lane line. The first guide lane line and the second guide lane line are the two guide lane lines that are farthest apart among the at least two guide lane lines.

10. The method according to claim 9, characterized in that, The step of obtaining the region of interest of the image to be processed based on the height information and the first region further includes: The length of the region of interest is obtained based on the length of the first line segment; The length of the object to be detected in the image to be processed is obtained based on the height information and the scale. The scale is used to indicate the proportional relationship between the length of the object to be detected in the image to be processed and the physical height of the object to be detected. The width of the region of interest is obtained based on the length of the object to be detected in the image to be processed.

11. The method according to any one of claims 1 to 5, 7 to 10, characterized in that, If the resolution of the region of interest obtained based on the height information and the first region is greater than a second preset threshold, the method further includes: The resolution of the region of interest is compressed to the second preset threshold.

12. The method according to any one of claims 1 to 5, 7 to 10, characterized in that, If the resolution of the region of interest obtained based on the height information and the first region is less than a second preset threshold, the method further includes: Super-resolution processing is performed on the region of interest to increase the resolution of the region of interest to the second preset threshold.

13. The method according to any one of claims 1 to 5, 7 to 10, characterized in that, The objects to be detected include traffic lights.

14. An image processing apparatus, characterized in that, include: The acquisition module is used to acquire the image to be processed; An image segmentation module is used to input the image to be processed into a first neural network to obtain a first prediction result; The region of interest module is further configured to, when the first prediction result indicates that the first region of the image to be processed is a lane line, obtain the region of interest of the object to be detected in the image to be processed based on the height information and the first region, wherein the height information includes a preset physical height of the object to be detected, and the region of interest is used by the second neural network to obtain candidate boxes and classification of the object to be detected; when the lane line in the first region includes a stop line, the region of interest module is specifically configured to: Obtain the length of the stop line in the image to be processed; The length of the region of interest is obtained based on the length of the stop line in the image to be processed; The length of the object to be detected in the image to be processed is obtained based on the height information and the scale. The scale is used to indicate the proportional relationship between the length of the object to be detected in the image to be processed and the physical height of the object to be detected. The width of the region of interest is obtained based on the length of the object to be detected in the image to be processed.

15. The image processing apparatus according to claim 14, characterized in that, The first region includes a plurality of first pixels, wherein the probability of each of the plurality of first pixels belonging to the stop line exceeds a first preset threshold, and the stop line is composed of the plurality of first pixels. The region of interest module is specifically used for: The length of the stop line in the image to be processed is obtained based on the distance between the two farthest pixels among the plurality of first pixels.

16. The image processing apparatus according to claim 14, characterized in that, The region of interest module is also used for: Obtain a first distance, which is the distance between the object to be detected and the vehicle; Obtain a second distance, which is the distance between the stop line and the lower edge of the image to be processed; The scale is obtained based on the first distance and the second distance.

17. The image processing apparatus according to claim 14, characterized in that, When the lane lines in the first region also include at least two guide lane lines, the region of interest module is further configured to: Obtain the width of any two adjacent guide lane lines in the image to be processed from the at least two guide lane lines; The scale is obtained based on the width of any two adjacent guide lane lines in the image to be processed and the preset physical width of the two guide lane lines.

18. The image processing apparatus according to claim 17, characterized in that, The region of interest module is specifically used for: The length of the region of interest is obtained based on the distance between the first intersection point and the second intersection point. The first intersection point is the intersection point of the first guide lane line and one end of the stop line in the image to be processed. The second intersection point is the intersection point of the second guide lane line and the other end of the stop line in the image to be processed. The first guide lane line and the second guide lane line are the two guide lane lines that are farthest apart among the at least two guide lane lines.

19. The image processing apparatus according to any one of claims 14 to 18, characterized in that, The position of the lower edge of the region of interest is determined based on the position of the stop line in the image to be processed.

20. The image processing apparatus according to claim 14, characterized in that, When the lane markings in the first region include at least two guide lane markings but do not include a stop line, the region of interest module is specifically used for: The length of the region of interest is obtained based on the distance between the third and fourth intersection points. The third intersection point is the intersection of one end of the first guide lane line and the first line segment in the image to be processed. The fourth intersection point is the intersection of the other end of the second guide lane line and the first line segment in the image to be processed. The first guide lane line and the second guide lane line are the two guide lane lines that are farthest apart among the at least two guide lane lines. The first line segment is a line segment that passes through a second pixel. The second pixel is the pixel corresponding to the highest point of the shortest guide lane line among the at least two guide lane lines in the image to be processed. The length of the object to be detected in the image to be processed is obtained based on the height information and the scale. The scale is used to indicate the proportional relationship between the length of the object to be detected in the image to be processed and the physical height of the object to be detected. The width of the region of interest is obtained based on the length of the object to be detected in the image to be processed.

21. The image processing apparatus according to claim 20, characterized in that, The first line segment is parallel to the lower edge of the image to be processed.

22. The image processing apparatus according to claim 14, characterized in that, When the lane lines in the first region include at least two guide lane lines but do not include a stop line, the position of the lower edge of the region of interest is determined according to the position of the first line segment in the image to be processed. The first line segment occupies a preset length of pixels, and one end of the first line segment intersects with the first guide lane line, and the other end of the first line segment intersects with the second guide lane line. The first guide lane line and the second guide lane line are the two guide lane lines that are farthest apart among the at least two guide lane lines.

23. The image processing apparatus according to claim 22, characterized in that, The region of interest module is specifically used for: The length of the region of interest is obtained based on the length of the first line segment; The length of the object to be detected in the image to be processed is obtained based on the height information and the scale. The scale is used to indicate the proportional relationship between the length of the object to be detected in the image to be processed and the physical height of the object to be detected. The width of the region of interest is obtained based on the length of the object to be detected in the image to be processed.

24. The image processing apparatus according to any one of claims 14 to 18, 20 to 23, characterized in that, It also includes a compression module. The compression module is used to compress the resolution of the region of interest to the second preset threshold if the resolution of the region of interest obtained based on the height information and the first region is greater than the second preset threshold.

25. The image processing apparatus according to any one of claims 14 to 18, 20 to 23, characterized in that, It also includes a super-resolution processing module. The super-resolution processing module is used to perform super-resolution processing on the region of interest if the resolution of the region of interest obtained based on the height information and the first region is less than a second preset threshold, so as to increase the resolution of the region of interest to the second preset threshold.

26. The image processing apparatus according to any one of claims 14 to 18, 20 to 23, characterized in that, The objects to be detected include traffic lights.

27. An image processing apparatus, characterized in that, The method includes a processor coupled to a memory storing program instructions, which, when executed by the processor, implement the method of any one of claims 1 to 13.

28. A computer-readable storage medium, characterized in that, Includes a program that, when run on a computer, causes the computer to perform the method as described in any one of claims 1 to 13.

29. A computer program product, characterized in that, When the computer program product is run on a computer, it causes the computer to perform the method described in any one of claims 1 to 13.

30. A chip, characterized in that, The chip is coupled to a memory for executing a program stored in the memory to perform the method as described in any one of claims 1 to 13.

31. An intelligent vehicle, characterized in that, The intelligent vehicle includes a processing circuit and a storage circuit, the processing circuit and the storage circuit being configured to perform the method as described in any one of claims 1 to 13.

Images