Computer system with visual front-end processing mechanism and method of operation thereof

Abstract

A computer system with a visual front-end processing mechanism and a method of operation thereof are disclosed. A computing system includes a control circuit configured to capture a sensor data stream provided by an optical sensor, extract a lane line from the sensor data stream, optimize external and internal parameters based on the extraction of the lane line, and assemble optimized parameters including the external and internal parameters; and a communication circuit, coupled to the control circuit, configured to receive an optical sensor alarm for display on the device.

Description

Technical Field

[0001] Embodiments of the present invention relate generally to computing systems, and more particularly to systems having visual simultaneous localization and mapping (SLAM) systems. Background Art

[0002] As autonomous vehicles emerge into the scene, they require detailed high-definition maps that can show every aspect of the road along the route and a network that can verify the features listed in the high-definition map. The optical network can consist of multiple cameras that observe all sides of the environment around the autonomous vehicle. For the safe operation of autonomous vehicles, a combination of optical networks and high-definition maps is required.

[0003] Therefore, there still exists a need for a computing system with a visual simultaneous localization and mapping (SLAM) mechanism to provide increased speed and lower overhead for converting optical sensor data streams into digitally processable information. Given the increasing competitive pressures of business, as well as growing consumer expectations and diminishing opportunities for meaningful product differentiation in the marketplace, finding answers to these questions is increasingly critical. In addition, the need to reduce costs, improve efficiency and performance, and meet competitive pressures adds even greater urgency to the critical need for finding answers to these questions.

[0004] Solutions to these problems have been long sought but prior developments have not taught or suggested any solutions and, thus, solutions to these problems have long eluded those skilled in the art. Summary of the invention

[0005] An embodiment of the present invention provides a method for operating a computing system, including: capturing a sensor data stream of an area of ​​interest provided by an optical sensor; correcting the optical distortion of the optical sensor for the area of ​​interest; extracting line segments from the Nth frame of the area of ​​interest; calculating the zero-mean normalized cross-correlation ZNCC score of the line segments in the Nth frame and the N+1th frame; identifying a feature line by the ZNCC score being greater than a ZNCC threshold; and calculating the GPS coordinates of the feature line to control a device.

[0006] An embodiment of the present invention provides a computing system, comprising a control circuit, wherein the control circuit is configured to: capture a sensor data stream of an area of ​​interest provided by an optical sensor, correct the optical distortion of the optical sensor for the area of ​​interest, extract line segments from the Nth frame of the area of ​​interest, calculate the zero-mean normalized cross-correlation ZNCC score of the line segments in the Nth frame and the N+1th frame, identify feature lines by the ZNCC score being greater than a ZNCC threshold, and calculate the GPS coordinates of the feature lines to control a device.

[0007] An embodiment of the present invention provides a non-transitory computer-readable medium comprising instructions for a computing system, including: capturing a sensor data stream of an area of ​​interest provided by an optical sensor; correcting optical distortion of the optical sensor for the area of ​​interest; extracting line segments from the Nth frame of the area of ​​interest; calculating the zero-mean normalized cross-correlation ZNCC score of the line segments in the Nth frame and the N+1th frame; identifying a feature line by the ZNCC score being greater than a ZNCC threshold; and calculating the GPS coordinates of the feature line to control a device.

[0008] Certain embodiments of the present invention have other steps or elements in addition to or in place of those described above.The steps or elements will become apparent to those skilled in the art by reading the following detailed description when referring to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 is a block diagram of a computing system with a visual simultaneous localization and mapping (SLAM) mechanism in an embodiment of the present invention.

[0010] Figure 2 is an example of a top plan view illustration of a vehicle for a computing system.

[0011] Figure 3 is an exemplary projection view processed by a computing system.

[0012] Figure 4 An exemplary control flow diagram of a computing system is shown.

[0013] Figure 5 is an exemplary block diagram of a computing system in an embodiment.

[0014] Figure 6 is an exemplary operation flow chart of a computing system in an embodiment of the present invention.

[0015] Figure 7 is an exemplary line identification mechanism in an embodiment.

[0016] Figure 8is an exemplary line identification diagram using neighboring lines in previous frames.

[0017] Fig. 9 is a flow chart of an operating method of a computing system in an embodiment of the present invention. DETAILED DESCRIPTION

[0018] The features described may be performed in combination on an individual basis without changing the claimed invention. The description of individual features is for clarity and understanding of the depth and breadth of the claimed invention, rather than limiting the combination of those features. Any combination of features may be performed.

[0019] The optical network may consist of four cameras with fisheye lenses that are strategically placed to observe all sides of the autonomous vehicle. The functionality of these cameras is critical to the safe operation of autonomous vehicles. As these vehicles age, external and internal parameters may deviate from the original values. The changes may be the result of oxidized lenses, dust or debris on the lens or housing of the camera, position changes on the vehicle body, or a combination of such elements. To account for real-time changes in the optical network, an automatic optical calibration mechanism can dynamically adjust the external and internal parameters and alert the user that a parameter defect has been detected.

[0020] The following embodiments are described in sufficient detail to enable those skilled in the art to make and use the invention. It should be understood that other embodiments will be apparent based on this disclosure, and that system, process or mechanical changes may be made without departing from the scope of the embodiments of the invention.

[0021] In the following description, many specific details are given to provide a thorough understanding of the present invention. However, it is apparent that the present invention can also be practiced without these specific details. In order to avoid obscuring the embodiments of the present invention, some well-known circuits, system configurations and process steps are not disclosed in detail.

[0022] The drawings showing embodiments of the system are semi-diagrammatic and not to scale, and in particular, some dimensions are for clarity of presentation and are shown exaggerated in the drawings. Similarly, although the views in the drawings generally show similar orientations for ease of description, the illustrations in the drawings are arbitrary in most cases. In general, the present invention can operate in any orientation. The embodiments of various components are for ease of description and are not intended to have any other meaning or provide limitations to the embodiments of the present invention.

[0023] Those skilled in the art will appreciate that the format used to express navigation information is not critical to some embodiments of the present invention. For example, in some embodiments, navigation information is presented in the format of (X, Y, Z); where X, Y, and Z are three coordinates defining a geographic location, i.e., the location of a vehicle, an artifact, or a pixel in an optical data frame. The basic function for converting optical feedback into a usable format with a high-definition map is a visual simultaneous localization and mapping (SLAM) mechanism.

[0024] The term "module" as used herein may be implemented as or include software running on dedicated hardware, hardware, or a combination thereof in the present invention, depending on the context in which the term "module" is used. For example, software may be machine code, firmware, embedded code, and application software. Software may also include functions, calls to functions, code blocks, or a combination thereof.

[0025] In addition, for example, hardware can be a gate, circuit, processor, computer, integrated circuit, integrated circuit core, memory device, pressure sensor, inertial sensor, micro-electromechanical system (MEMS), passive device, physical non-transitory storage medium including instructions for executing software functions, part thereof, or a combination thereof for controlling one or more hardware units or circuits. In addition, if "unit" is written in the system claim section below, "unit" is deemed to include hardware circuits for the purpose and scope of the system claim.

[0026] The units in the following description of the embodiments may be coupled or attached to each other as described or shown. The coupling or attachment may be direct or indirect, with no or intermediate items between the coupled or attached modules or units. The coupling or attachment may be performed by physical contact between the modules or units or by communication (such as wireless communication) between the modules or units.

[0027] The term "extrinsic parameters" as used in this specification refers to the pose of the camera being analyzed. The pose of the camera defines the physical relationship between the camera and the real-world object that the camera is viewing. The variables used to analyze the pose are direction, pitch, and yaw. The term "intrinsic parameters" as used in this specification refers to the physical parameters within the camera, including focal length, lens distortion coefficients, zoom factors, principal point location, and pixel size. The combination of extrinsic and intrinsic parameters defines the physical transformation of the real-world view to the rectified captured image within the camera.

[0028] As used in the specification and claims, the term "fuse" or "fusing" has the meaning of combining, connecting, merging, or integrating as appropriate to the use of the term. In the case of an optical network, fusing the views of adjacent cameras can identify elements displayed in more than one view from adjacent cameras.

[0029] It should also be understood that the nouns or elements in the embodiments may be described as singular instances. It should be understood that the use of the singular is not limited to the singular, but rather the use of the singular may apply to multiple instances of any particular noun or element in the present application. Many instances may be the same or similar, or may be different.

[0030] Reference now Figure 1 , which shows a block diagram of a computing system 100 with a visual simultaneous localization and mapping (SLAM) mechanism in an embodiment of the present invention. The computing system 100 may include a first device 102 such as a client or a server, and the first device 102 is connected to a second device 106 such as a client or a server.

[0031] The computing system 100 may include a system for capturing detailed information about the current environment based on a fusion of multiple sources to coordinate and quickly identify specific details about the driving area. The first device 102 may communicate with the second device 106 through a network 104 such as a wireless or wired network.

[0032] For example, the first device 102 can be any of a variety of computing devices, such as a cellular phone, a personal digital assistant, a notebook computer, an autonomous vehicle, an automotive telematics navigation system, or other multi-function device. In addition, for example, the first device 102 can include a device or subsystem, an autonomous or self-steering vehicle or object, a driver-assisted vehicle, a remotely controlled vehicle or object, or a combination thereof.

[0033] The first device 102 may be coupled directly or indirectly to the network 104 to communicate with the second device 106, or may be a standalone device. The first device 102 may further be separate from or incorporated into a vehicle, such as a car, truck, bus, or motorcycle.

[0034] For illustrative purposes, the computing system 100 is described with the first device 102 as a mobile computing device, but it should be understood that the first device 102 can be different types of devices. The first device 102 can be an autonomous vehicle or an accessory device coupled to a telematics vehicle network to support an autonomous vehicle.

[0035] The second device 106 can be any of a variety of centralized or decentralized computing devices. For example, the second device 106 can be a computer, a grid computing resource, a virtualized computer resource, a cloud computing resource, a router, a switch, a peer-to-peer distributed computing device, or a combination thereof.

[0036] The second device 106 can be centralized in a single room, distributed in different rooms, distributed in different geographical locations, embedded in a telecommunications network. The second device 106 can be coupled to the network 104 to communicate with the first device 102. The second device 106 can also be a client type device as described for the first device 102.

[0037] For illustrative purposes, the computing system 100 is described with the second device 106 as a non-mobile computing device, but it should be understood that the second device 106 can be a different type of computing device. For example, the second device 106 can also be a mobile computing device, such as a notebook computer, another client device, a wearable device, or a different type of client device.

[0038] Furthermore, for the purpose of illustration, the computing system 100 is described with the second device 106 as a computing device, but it should be understood that the second device 106 can be a different type of device. Furthermore, for the purpose of illustration, the computing system 100 is shown as having the second device 106 and the first device 102 as endpoints of the network 104, but it should be understood that the computing system 100 can include different partitions between the first device 102, the second device 106, and the network 104. For example, the first device 102, the second device 106, or a combination thereof can also act as part of the network 104.

[0039] The network 104 can span and represent a variety of networks. For example, the network 104 can include wireless communication, wired communication, light, ultrasound, or a combination thereof. Satellite communication, cellular communication, Bluetooth, infrared data association standard (IrDA), wireless fidelity (WiFi), and world microwave access interoperability (WiMAX) are examples of wireless communications that can be included in the communication path. Ethernet, digital subscriber line (DSL), fiber to the home (FTTH), and plain old telephone service (POTS) are examples of wired communications that can be included in the network 104. In addition, the network 104 can span multiple network topologies and distances. For example, the network 104 can include a direct connection, a personal area network (PAN), a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), or a combination thereof.

[0040] The computing system 100 may provide additional features not available in computing systems of the prior art. The first device 102 may be coupled to an optical sensor 108 including a sensor data local storage 110. The optical sensor 108 is a monitoring device, such as a set of monocular cameras disposed on the first device 102, configured to monitor, observe, record, or a combination thereof, the surrounding environment of the first device 102. The sensor data local storage 110 provides a non-transitory storage medium loaded by a distortion correction unit 112 to store a sensor data stream 111 captured by the optical sensor 108. The distortion correction unit 112 may be a hardware structure that provides machine learning and may include four sets of memories that may be loaded from the sensor data stream 111 of the optical sensor 108 for manipulating the sensor data stream 111 in order to identify and correct optical distortion from the optical sensor 108. The distortion correction unit 112 may include circuitry that analyzes overlapping areas from each optical sensor 108 to identify items or elements that appear in more than one optical sensor 108 in order to identify optical distortion 113 of the optical sensor 108.

[0041] The distortion correction unit 112 may be coupled to a zero mean normalized cross correlation (ZNCC) tracking unit 114 in the sensor data local storage 110, which is capable of analyzing fixed-size frames from each optical sensor 108, such as frames captured periodically, and the sensor data stream 111 may be analyzed by a fast line detection unit 116 that is sequentially coupled to the ZNCC tracking unit 114. The ZNCC tracking unit 114 may be a hardware structure that receives the output of the fast line detection unit 116, such as matching line pairs between consecutive frames of the sensor data stream 111. The ZNCC tracking unit 114 also receives GPS coordinates 115 from the first device 102. Once processed, the ZNCC tracking unit 114 may serially correct the GPS coordinates 115 of the lines and features in the sensor data stream 111. It should be understood that the GPS coordinates 115 may include the latitude, longitude, and altitude of a single point or pixel in the sensor data stream 111.

[0042] For example, the sensor data local storage device 110 can be implemented in a variety of ways, such as a non-volatile storage device (such as a hard drive, a solid-state storage device (SSD), a flash memory card, or a combination thereof), which is capable of fusing the sensor data stream 111 from each optical sensor 108 into the fast line detection unit 116 for further analysis by the ZNCC tracking unit 114.

[0043] The distortion correction unit 112 may provide a pixel evaluation module, which may be implemented in software running on dedicated hardware, full hardware, or a combination thereof, configured to analyze the photographic scene to identify features and lines around the first device 102. The data may include road markings, lane lines, road boundaries, traffic control devices, crosswalks, lane markings, buildings, physical structures, signs, lighting, or a combination thereof. The distortion correction unit 112 may process the sensor data stream 111 including sampled data frames from each optical sensor 108 to provide the relative position of the features to the ZNCC tracking unit 114 based on the physical position of the optical sensor 108.

[0044] The ZNCC tracking unit 114 may process the sensor data stream 111 to identify the geographic locations of lines and features in the sensor data stream 111. The fast line detection unit 116 may identify lines and shapes from the sensor data stream 111 of roads in the area around the first device 102. It should be understood that the compilation of the sensor data streams 111 from the four optical sensors 108 may be analyzed by the fast line detection unit 116. The fast line detection unit 116 is a hardware structure supported by software that may identify lines and shapes in the view of the area around the first device 102. The fast line detection unit 116 provides a fusion of the sensor data streams 111 from each optical sensor 108, including identifying any overlapping views of features observed in more than one optical sensor 108. The distortion correction unit 112 may provide position corrections for features and elements that are duplicated in more than one sensor data stream 111. It will be understood that the wide angle lens of the optical sensor 108 will provide overlapping areas of viewing the same object from different angles. Distortion correction unit 112 may correct pose errors caused by these overlapping regions during fusion of sensor data streams 111. Fast line detection unit 116 may handle the fusion.

[0045] The sensor data local storage 110 may be coupled or integrated with the optical sensor 108 to store a sensor data stream 111 from each optical sensor 108 available in the first device 102. The first device 102 may access a distortion correction unit 112 to adjust the relative performance of the optical sensors 108. The distortion correction unit 112 may be trained to recognize and adjust internal parameters of the optical sensors 108. The distortion correction unit 112 may recognize and correct optical distortion 113 to pass the corrected contents of the sensor data local storage 110 to the ZNCC tracking unit 114 for feature tracking between consecutive frames of the sensor data stream 111.

[0046] The computing system 100 may operate without a user 118. The user 118 may include an individual or entity accessing or utilizing the computing system 100 or a device therein. For example, the user 118 may include an individual who owns or operates the first device 102, a service, or a combination thereof. In addition, for example, the user 118 may access or utilize the second device 106 through the first device 102, a service, or a combination thereof. The second device 106 may access a high-definition map 120 to provide detailed route information to the first device 102. The second device 106 may provide a traffic flow status 109 to the first device 102 to provide additional information for controlling the first device 102. The traffic flow status 109 may be a report or list of vehicles or obstacles in an area traveled by the first device 102.

[0047] The computing system 100 can further process direct user input 119 from the user 118. The direct user input 119 can include the user 118 taking manual control of the autonomous vehicle or the first device 102. The direct user input 119 can be provided by the user 118 directly to the first device 102 or directly on the first device 102.

[0048] Reference now Figure 2 , which shows the Figure 1 An example of a top plan view 201 of a vehicle of a computing system 100 is illustrated. The computing system 100 may be included in or interact with the first device 102. Because the setup and calibration of the optical network is performed by a specific process when the autonomous vehicle is first manufactured. The degradation of the optical network must be verified regularly by the manufacturer during planned maintenance. The possible degradation of the optical network between planned maintenance may pose a significant hazard to passengers and the public. Figure 1 The distortion correction unit 112 can identify and adjust Figure 1 The distortion correction unit 112 may also alert the user 118 when a particular one of the optical sensors 108 cannot be adjusted correctly.

[0049] The first device 102 may be an object or machine for transporting people or goods, and may be capable of automatically manipulating or operating the object or machine. Figure 1 The first device 102 may include a vehicle accessed by the user 118 to control, manipulate, operate, or a combination thereof. For example, the first device 102 may include a car, a truck, a cart, a drone, or a combination thereof.

[0050] Can be used without corresponding manipulation or movement Figure 1The first device 102 may be further controlled or manipulated without direct user input 119. For example, the first device 102 may include a self-driving vehicle, or a vehicle with automatic maneuvering features (such as smart cruise control or preventive braking). The first device 102 may include a smart cruise control feature that is capable of setting and adjusting the driving speed of the first device 102 without direct user input 119. In addition, for example, the first device 102 may be controlled or manipulated by the computing system 100, including the computing system 100 controlling or setting the cruising speed, lane position, or other physical manipulation or movement of the first device 102.

[0051] The computing system 100 may further employ a distortion correction unit 112 from one or more vehicles or devices. The distortion correction unit 112 may provide important reliability information and adjustments for the front optical sensor 220, the rear optical sensor 222, the left optical sensor 224, and the right optical sensor 226. Each of the front optical sensor 220, the rear optical sensor 222, the left optical sensor 224, and the right optical sensor 226 may be a hardware camera including a fisheye lens 221 having a 170 degree viewing angle of the region of interest 203.

[0052] The first device 102 or other vehicle interfaced with the computing system 100 may include devices, circuits, one or more specific sensors (such as environmental sensors 210), or a combination thereof, for providing auxiliary or additional information to the user 118 controlling, manipulating, or operating the first device 102. The environmental sensor 210 may include a cabin camera, a LiDAR sensor, an optical sensor 108, or a combination thereof. The optical sensor 108 may be a camera with a wide-angle lens, such as a fisheye lens 221. The optical sensor 108 may be mounted on the exterior of the first device 102 disposed at the front, rear, left, and right sides of the first device 102. The first device 102 or any other vehicle may include vehicle communication circuitry 204, vehicle control circuitry 206, vehicle storage circuitry 208, other interfaces, or a combination thereof.

[0053] The vehicle storage circuit 208 may include a functional unit or circuit that is integrated into the corresponding first device 102 and is configured to store and recall information. The vehicle storage circuit 208 may be a volatile memory, a non-volatile memory, an internal memory, an external memory, or a combination thereof, and may include Figure 1 The sensor data local storage device 110. For example, the vehicle storage circuit 208 can be a non-volatile storage device (such as non-volatile random access memory (NVRAM), flash memory, disk storage device), or a volatile storage device (such as static random access memory (SRAM)).

[0054] The vehicle storage circuit 208 may store vehicle software, other relevant data such as input information, information from sensors, processing results, information predetermined or preloaded by the computing system 100 or the vehicle manufacturer, or a combination thereof.

[0055] The vehicle control circuit 206 may include functional units or circuits that are integrated into the first device 102 and configured as a processor to execute or implement instructions. The vehicle control circuit 206 may execute or implement vehicle software to provide intelligence corresponding to the vehicle, computing system 100, or a combination thereof.

[0056] The vehicle control circuit 206 can be implemented in a variety of different ways. For example, the vehicle control circuit 206 can be a processor, an application specific integrated circuit (ASIC), an embedded processor, a microprocessor, hardware control logic, a hardware finite state machine (FSM), a digital signal processor (DSP), or a combination thereof. As a more specific example, the vehicle control circuit 206 can include an engine control unit, one or more central processing units, or a combination thereof.

[0057] The vehicle communication circuit 204 may include a functional unit or circuit that is integrated into a corresponding vehicle, such as the first device 102, another vehicle, or a combination thereof. The vehicle communication circuit 204 may be configured to enable external communications to and from the corresponding vehicle. For example, the vehicle communication circuit 204 may allow the first device 102 to communicate with the vehicle via a Figure 1 The network 104 communicates with the second device 106.

[0058] The vehicle communication circuit 204 may also function as a communication hub that allows a corresponding control vehicle to function as part of the network 104 and is not limited to being an endpoint or terminal circuit of the network 104. The vehicle communication circuit 204 may include active and passive components, such as microelectronics, circuits, or antennas for interacting with the network 104. For example, the vehicle communication circuit 204 may include a modem, a transmitter, a receiver, a port, a connector, or a combination thereof for conducting wired communications, wireless communications, or a combination thereof.

[0059] The vehicle communication circuit 204 can be coupled to the network 104 to directly send or receive information between the vehicle communication circuit 204 and the second device 106 as the endpoint of the communication, such as for direct line-of-sight communication or point-to-point communication. The vehicle communication circuit 204 can be further coupled to the network 104 to send or receive information through a server or another intermediate device between the endpoints of the communication.

[0060] The first device 102 or other vehicle may further include a variety of interfaces. The first device 102 may include one or more interfaces for interaction or internal communication between functional units or circuits of the first device 102. For example, the first device 102 may include one or more interfaces for vehicle storage circuit 208, vehicle control circuit 206, or a combination thereof, such as a driver, firmware, wire connection or bus, protocol, or a combination thereof.

[0061] The first device 102 or other vehicle may further include one or more interfaces for interacting with an occupant, operator or passenger, user 118, or a combination thereof, relative to the corresponding vehicle. For example, the first device 102 or other vehicle may include a user interface 212 including an input or output device or circuit, such as a screen or touch screen, a speaker, a microphone, a keyboard or other input device, a dashboard, or a combination thereof.

[0062] The first device 102 may further include one or more interfaces and switches or actuators for physically controlling movable components of the first device 102. For example, the first device 102 may include one or more interfaces and control mechanisms for physically performing and controlling manipulation of the first device 102, such as for autonomous driving, smart cruise control, or a steering feature.

[0063] Functional units or circuits in the first device 102 can operate individually and independently of other functional units or circuits. The first device 102 can operate individually and independently of the network 104, the second device 106, other devices or vehicles, or a combination thereof.

[0064] The above functional units or circuits may be implemented in hardware. For example, one or more functional units or circuits may be implemented using gates, circuits, processors, computers, integrated circuits, integrated circuit cores, pressure sensors, inertial sensors, micro-electromechanical systems (MEMS), passive devices, physical non-transitory storage media containing instructions for executing software functions, a portion thereof, or a combination thereof.

[0065] The environmental sensors 210 are each a device or circuit for detecting or identifying the environment of the corresponding vehicle. The environmental sensors 210 can detect, identify, determine, or a combination thereof, such as the state, surrounding environment, or movement of the corresponding vehicle. The environmental sensors 210 can detect, identify, determine, or a combination thereof, the environment inside the cabin of the corresponding vehicle, the environment outside and around the corresponding vehicle, or a combination thereof. The environmental sensors 210 can be implemented for the first device 102.

[0066] For example, the environmental sensor 210 may include a user interface 212, an optical sensor 108, a radar sensor 216, a position movement sensor 218, or a combination thereof. The user interface 212 may include a projector, a video screen, a touch screen, a speaker, or any combination thereof. The user interface 212 may display a planned route, lane suggestions, speed warnings, vehicle system alerts, and combinations thereof.

[0067] The optical sensor 108 may include a sensor for detecting or determining visual information representing the environment outside and around the corresponding vehicle. The optical sensor 108 may include a camera attached to or integrated with the corresponding vehicle or device. For example, the optical sensor 108 may include a camera, such as a front camera, a video camera, a rear view or backup camera, a side view or blind spot camera, or a combination thereof. In addition, for example, the environmental sensor 210 may include an infrared sensor, a night vision camera, or a night vision sensor.

[0068] The optical sensor 108 may further include a camera on the first device 102 or another user device of the user 118 connected to and interacting with the vehicle. The optical sensor 108 may further include a cabin camera for detecting or determining visual information of the vehicle or the interior of the cabin of the vehicle.

[0069] Radar sensor 216 may include an object detection system, device, or circuit. Radar sensor 216 may determine or identify the presence of an object or target (such as an obstacle or another vehicle) outside the corresponding device or vehicle, the relative position or distance between the object or target and the corresponding device or vehicle, or a combination thereof.

[0070] The radar sensor 216 can use radio waves to determine or identify the presence of an object or target, a relative position or distance relative to the first device 102 or other corresponding devices or vehicles, or a combination thereof. For example, the radar sensor 216 can include a proximity sensor or warning system, such as for an area geographically or physically relative to the front, rear, adjacent, or side of the first device 102, or a combination thereof.

[0071] The position movement sensor 218 may be a sensor for identifying or calculating the geographic location of the corresponding vehicle or device, determining the movement or speed of the corresponding vehicle or device, or a combination thereof. The position movement sensor 218 may include an accelerometer, a speedometer, a global positioning system (GPS) receiver or device, a gyroscope, or a compass, or a combination thereof. In addition to the position movement sensor 218 or in addition to the position movement sensor 218, the first device 102 may include an environmental sensor 210. The position movement sensor 218 may provide a gyroscope rate of change for monitoring turns and a speed from a speedometer.

[0072] The computing system 100 may use one or more optical sensors 108 corresponding to one or more devices, one or more vehicles, or a combination thereof to generate a traffic flow state 109 that describes or represents information about the environment surrounding the corresponding device or vehicle. Figure 1 The sensor data stream 111 may be further processed by the vehicle control circuitry 206 , stored in the vehicle storage circuitry 208 , transmitted to another device or vehicle via the vehicle communication circuitry 204 , or a combination thereof.

[0073] As a more specific example, the vehicle communication circuit 204 , the vehicle control circuit 206 , the vehicle storage circuit 208 , the optical sensor 108 , one or more interfaces, or a combination thereof may be included in or constitute the first device 102 .

[0074] The computing system 100 can provide the vehicle movement control 228 as a suggestion to the user 118 for maneuvering or operating the first device 102. Details regarding the use and processing of the vehicle movement control 228 are discussed below.

[0075] The computing system 100 can process and generate vehicle movement controls 228 for controlling or manipulating the first device 102. The vehicle movement controls 228 are instructions, signals, processes, methods, mechanisms, or combinations thereof for directing or controlling the physical movement or travel of the first device 102.

[0076] Continuing with this example, the computer system 100 can use the vehicle movement control 228 provided from the first device 102 without requiring the user input 119. The computing system 100 can employ the vehicle movement control 228 to provide information, assist maneuvers, control maneuvers, or a combination thereof for other vehicles near the first device 102.

[0077] Continuing with this example, the computing system 100 can transmit the traffic flow state 109 to other devices or vehicles through the second device 106, or directly to other devices or vehicles, such as for a point-to-point communication system. The computing system 100 can transmit the traffic flow state 109 to inform other devices or vehicles of the state of the first device 102 itself, the state of other vehicles detected and identified around the first device 102, or a combination thereof.

[0078] As a more specific example, the computing system 100 can use the traffic flow state 109 to detect weaknesses in the optical sensor 108 that would prevent safe operation of the vehicle movement control 228, such as for steering, braking, setting or adjusting travel speed, accessory control, or a combination thereof. Details regarding the processing of the vehicle movement control 228 are discussed below.

[0079] It should be understood that the optical sensors 108 can be disposed at fixed locations around the first device 102. For example, the front optical sensor 220, the rear optical sensor 222, the left optical sensor 224, and the right optical sensor 226 can monitor the area of ​​interest 203 including details of the road 202. The information from each of the front optical sensor 220, the rear optical sensor 222, the left optical sensor 224, and the right optical sensor 226 can be displayed. Figure 1 The combination of sensor data streams 111 can be composed of Figure 1 The distortion correction unit 112 performs fusion to form a surrounding view of the region of interest 203.

[0080] It should be understood that due to the characteristics of the wide-angle lens and the relative position of the optical sensor 108, an object may appear in more than one optical data stream 111. For example, the fisheye view 230 may generate overlapping regions 232 at the four corners of the region of interest 203. It should be understood that the fisheye view 230 may be substantially equal to a 170 degree viewing angle. The resulting overlapping regions 232 may have a resolution of 170 degrees. Figure 1 For example, the traffic control sign 234 may be seen by both the front optical sensor 220 and the right optical sensor 226. The distortion correction unit 112 may resolve the traffic control sign 234 into a single unit of the traffic control sign 234 adjacent to the road 202.

[0081] Reference now Figure 3 , which shows an exemplary projection view 301 processed by the computing system 100. Figure 2 The region of interest 203 may be represented in the Nth frame 302 and the N+1th frame 304, which are obtained by the first device 102. Figure 1 The area viewed by the optical sensor 108. For example, Figure 2 The front optical sensor 220, Figure 2 The rear optical sensor 222, Figure 2 The left optical sensor 224 and Figure 2 The right optical sensor 226 may be combined in Figure 1 The sensor data of the local storage device 110 is stored in order to form the Nth frame 302 and the N+1th frame 304. The front optical sensor 220 may be a Figure 2 The front optical sensor 220 can provide a wide-angle view that can provide Figure 1 The sensor data stream 111. Figure 1 The fast line detection unit 116 may process both the Nth frame 302 and the N+1th frame 304 to identify the feature lines 306 in the Nth frame 302 and the N+1th frame 304 .

[0082] The principal point 308 may represent the center and undistorted point in the Nth frame 302 and the N+1th frame 304. As the distance from the principal point 308 increases in all directions, Figure 1 By assuming a specific relationship between the feature lines 306 extracted from the region of interest 203, the optical distortion 113 can be calculated by Figure 1 The optical distortion 113 may be determined by the distortion correction unit 112 of the front optical sensor 220. It should be understood that most of the internal parameters of the front optical sensor 220 are fixed at the time of manufacture. The optical distortion 113 may be a variable based on the condition of the fisheye lens 221. The distortion correction unit 112 may correct the optical distortion 113 in preparation for the fast line detection unit 116 to identify the feature line 306.

[0083] It has been found that the distortion correction unit 112 can learn correction factors through training and daily use, which correction factors include corrections for the characteristic line 306. The corrected distance measured in pixels can be provided to Figure 1 The fast line detection unit 116 is used for fast analysis and tracking by the ZNCC tracking unit 114.

[0084] Reference now Figure 4 , which shows Figure 1 401 of a computing system 100. The exemplary control flow diagram 401 shows a block 402 of an image from a sensor, which includes Figure 1 The distortion correction unit 112 processes the Figure 1 The optical sensor 108 Figure 1 Correction and fusion of sensor data stream 111. Figure 2 The view of the region of interest 203 may be corrected by the distortion correction unit 112 using machine learning or Figure 1 The fast line detection unit 116 is processed to extract edge lines from the optical data stream 111. The machine learning analysis of the distortion correction unit 112 can identify and correct the edge lines caused by Figure 2 Optical distortion 113 caused by fisheye lens 221.

[0085] It will be appreciated that the optical data stream 111 will show optical distortion 113. Figure 3 The computing system 100 knows that Figure 2 Any feature line 306 identified in the overlap region 232 is the same actual feature line in the feature lines 306. This information allows determination Figure 2 The optical distortion 113 of the fisheye lens 221 is reduced, and the distortion correction unit 112 is allowed to remove the influence of the optical distortion 113 from the sensor data stream 111.

[0086] The flow may proceed to block 404 of extracting lines from the undistorted image. The fast line detection unit 116 may separate and identify the lines formed by Figure 1 The optical sensor 108 is Figure 2 The fast line detection unit 116 may extract line segments 405 starting from the pixel edge to connect the straight line with the adjacent line segments in the line segments 405, and continue fitting the line and extending to the next edge pixel until it satisfies the co-linearity with the current line segment in the line segments 405. This process may continue until all line segments 405 have been extracted.

[0087] Flow may proceed to block 406 where sampling points for each line are determined. The fast line detection unit 116 may identify the relative positions of features and lines captured from the optical data stream 111. The fast line detection unit 116 may separate the lines by applying sampling points to each identified line. The sampling points may be applied by placing the sampling points at the start point, the end point, and evenly spaced along the line. The sampling points may be analyzed and compared to further define the line.

[0088] The flow may proceed to block 408 to find neighboring lines in the nearest frame for further analysis. The fast line detection unit 116 may further characterize the line by adding a sampling point at the center of the line and defining an angle θ between the line and the horizontal plane. Figure 3 The fast line detection unit 116 can check the extracted feature lines in the feature lines 306. Figure 3 The Nth frame 302 and Figure 3 The N+1th frame 304 is used to find hints of a solution.

[0089] Flow proceeds from block 406 of determining the sampling points for each line to block 410 of obtaining patches around the sampling points. Block 410 of obtaining patches around the sampling points may collect an array of pixels around each sampling point applied to the extracted feature lines in feature lines 306. The size of the array may be determined by Figure 1 The first device 102 uses Figure 1 The version of the optical sensor 108 of the invention is established by machine learning. For example, the size of the pixel patch can be a pixel array with a size of 7X7, 5X5, 3X3, or some other symmetrical array where the X dimension is equal to the Y dimension.

[0090] Flow may proceed from block 408 of finding neighboring lines in the most recent frame to block 412 of obtaining patches of neighboring lines. Block 412 of obtaining patches of neighboring lines receives input from block 408 of finding neighboring lines in the most recent frame and block 410 of obtaining patches near sampling points. Block 412 of obtaining patches of neighboring lines may compare and align neighboring lines with patches of pixels identified for feature lines 306.

[0091] The flow proceeds to block 414 where the similarity of corresponding patches is calculated to calculate the similarity of the pixel patches identified in block 412 where the patches of neighboring lines are obtained. The similarity of the pixel patches can be calculated as follows:

[0092]

[0093] Where P' is the selected pixel patch of the current frame, P is the pixel patch from the previous frame, X' and Y' are the X and Y positions of the pixel being analyzed in the current frame, and X" and Y" are the X and Y positions of the pixel being analyzed in the previous frame.

[0094] The flow then proceeds to block 416 where the matching patch of 2 lines is calculated by ZNCC, where the ZNCC tracking unit 114 may calculate the matching score of the points in the pixel patch. The ZNCC tracking unit 114 may perform the following matrix math operation:

[0095] ZNCC stands for Zero Mean Normalized Cross Correlation. The equation is as follows:

[0096] The average gray value is:

[0097]

[0098] Where (x, y) represents the position of the point in the image. 2n+1 represents the patch size. Img(x, y) represents the grayscale value of the pixel at (x, y).

[0099] The standard deviation is:

[0100]

[0101] ZNCC is defined as

[0102] Below, P1 represents (x1, y1, n) and P2 represents (x2, y2, n), where P1 and P2 should have the same patch size 2n+1.

[0103] If the value of ZNCC(P1, P2) is greater than a first threshold, the patches are considered similar patches. If the matching patches of the Nth frame 302 and the N+1th frame 304 are greater than a second threshold, the line is considered a good match.

[0104] Flow proceeds to block 418 of selecting the best matching line to identify all matching lines in the Nth frame 302 and the N+1th frame 304, including identifying the matching lines based on the position of the optical sensor 108 and the relative position of the feature line 306. Figure 1 Their GPS coordinates are 115.

[0105] Reference now Figure 5 , which shows an exemplary block diagram of a computing system 100 in an embodiment. The computing system 100 may include a first device 102, a network 104, and a second device 106. The first device 102 may send information to the second device 106 via the network 104 in a first device transmission 508. The second device 106 may send information to the first device 102 via the network 104 in a second device transmission 510.

[0106] For purposes of illustration, the computing system 100 is shown with the first device 102 being a client device, but it is understood that the computing system 100 may include the first device 102 as a different type of device. For example, the first device 102 may be a server including a first display interface 530 coupled to the user interface 212.

[0107] In addition, for the purpose of illustration, the computing system 100 is shown with the second device 106 as a server, but it should be understood that the computing system 100 can include the second device 106 as a different type of device. For example, the second device 106 can be a client device. For example, the computing system 100 can be implemented entirely on the first device 102. The second device 106 can provide additional computing speed and capabilities.

[0108] In addition, for the purpose of illustration, the computing system 100 is shown as having an interaction between a first device 102 and a second device 106. However, it should be understood that the first device 102 can be part or all of an autonomous vehicle, an intelligent vehicle, or a combination thereof. Similarly, the second device 106 can similarly interact with the first device 102 representing an autonomous vehicle, an intelligent vehicle, or a combination thereof.

[0109] For simplicity of description in this embodiment of the invention, the first device 102 will be described as a client device and the second device 106 will be described as a server device. Embodiments of the invention are not limited to this selection of device types. This selection is an example of an embodiment of the invention.

[0110] The first device 102 may include a first control circuit 512, a first storage circuit 514, a first communication circuit 516, a first interface circuit 518, and a first positioning circuit 520. The first control circuit 512 may include a first control interface 522. The first control circuit 512 may execute first software 526 to provide intelligence for the computing system 100.

[0111] The first control circuit 512 can be implemented in many different ways. For example, the first control circuit 512 can be a processor, an application specific integrated circuit (ASIC), an embedded processor, a microprocessor, a hardware control logic, a hardware finite state machine (FSM), a digital signal processor (DSP) or a combination thereof. The first control interface 522 can be used for communication between the first control circuit 512 and other functional units or circuits in the first device 102. The first control interface 522 can also be used for communication outside the first device 102.

[0112] The first control interface 522 can receive information from other functional units / circuits or from external sources, or can send information to other functional units / circuits or to external destinations. External sources and external destinations refer to sources and destinations external to the first device 102.

[0113] The first control interface 522 can be implemented in different ways and can include different implementations, depending on which functional units / circuits or external units / circuits are being interfaced with the first control interface 522. For example, the first control interface 522 can be implemented using pressure sensors, inertial sensors, micro-electromechanical systems (MEMS), optical circuits, waveguides, wireless circuits, wired circuits, analog circuits, or combinations thereof.

[0114] The first storage circuit 514 may store the first software 526. The first storage circuit 514 may also store related information, such as data representing incoming images, data representing previously presented images, sound files, or a combination thereof.

[0115] The first storage circuit 514 may be a volatile memory, a non-volatile memory, an internal memory, an external memory, or a combination thereof. For example, the first storage circuit 514 may be a non-volatile storage device (such as a non-volatile random access memory (NVRAM), a flash memory, a disk storage device) or a volatile storage device (such as a static random access memory (SRAM)).

[0116] The first storage circuit 514 may include a first storage interface 524. The first storage interface 524 may be used to connect the first storage circuit 514 to other functional units or circuits in the first device 102 (such as Figure 1 The first storage interface 524 can also be used for communication outside the first device 102.

[0117] The first storage interface 524 can be a hardware circuit that is configured to receive information from other functional units / circuits or from external sources, or can send information to other functional units / circuits or to external destinations. External sources and external destinations refer to sources and destinations external to the first device 102.

[0118] The first storage interface 524 can include different implementations, depending on which functional units / circuits or external units / circuits are interfaced with the first storage circuit 514. The first storage interface 524 can be implemented using technologies and techniques similar to the implementations of the first control interface 522, such as analog circuits, digital circuits, wireless circuits, or wired circuits.

[0119] The first communication circuit 516 can enable external communications to and from the first device 102. For example, the first communication circuit 516 can allow the first device 102 to communicate with the second device 106 and the network 104.

[0120] The first communication circuit 516 may also function as a communication hub, thereby allowing the first device 102 to function as part of the network 104 and is not limited to functioning as an endpoint or terminal circuit for the network 104. The first communication circuit 516 may include active and passive components, such as microelectronics, analog circuits, wireless circuits, wired circuits, or antennas, for interacting with the network 104.

[0121] The first communication circuit 516 may include a first communication interface 528. The first communication interface 528 may be used for communication between the first communication circuit 516 and other functional units or circuits in the first device 102. The first communication interface 528 may receive information from the second device 106 to distribute to other functional units / circuits, or may send information to other functional units or circuits.

[0122] The first communication interface 528 may include different implementations, depending on which functional units or circuits are interfaced with the first communication circuit 516. The first communication interface 528 may be implemented using technologies and techniques similar to the implementations of the first control interface 522, including micro-electromechanical systems (MEMS), optical circuits, waveguides, wireless circuits, wired circuits, analog circuits, or combinations thereof.

[0123] The first interface circuit 518 allows Figure 1 The user 118 of the first device 102 interfaces and interacts. The first interface circuit 518 may include an input device and an output device. Examples of input devices of the first interface circuit 518 may include a keypad, a touchpad, soft keys, a keyboard, a microphone, an infrared sensor for receiving remote signals, an optical sensor 108, or any combination thereof to provide data and communication input. For example, the optical sensor 108 may be connected to the first interface circuit 518 via a wired or wireless connection. The first interface circuit 518 can pass the input from the optical sensor 108 to the first control circuit 512 for processing and storage.

[0124] The first interface circuit 518 may include a first display interface 530. The first display interface 530 may include an output device. The first display interface 530 may be coupled to the user interface 212, including a projector, a video screen, a touch screen, a speaker, a microphone, a keyboard, and combinations thereof.

[0125] The first control circuit 512 may also execute first software 526 for other functions of the computing system 100, including receiving location information from the first positioning circuit 520. The first control circuit 512 may further execute the first software 526 to interact with the network 104 via the first communication circuit 516.

[0126] The first control circuit 512 can operate the first interface circuit 518 to collect the sensor data stream 111 from the optical sensor 108. The first control circuit 512 can also receive position information from the first positioning circuit 520. The first control circuit 512 can operate Figure 1 The distortion correction unit 112 is configured to provide the ZNCC tracking unit 114 with the Figure 2 The indicator of the overlap area 232 of the sensor data stream 111 is used to further process and correct the sensor data stream 111. Figure 1 Optical distortion 113.

[0127] The first positioning circuit 520 may be a hardware circuit configured to generate position information for identifying real-world coordinates, such as Figure 1 The first control circuit 512 can calculate the geographic location of the first device 102 determined by the positioning circuit 520, including the geographic location of the optical sensor 108. The first positioning circuit 520 can be implemented in many ways. For example, the first positioning circuit 520 can be used as at least a part of a global positioning system, an inertial computing system, a cellular tower positioning system, a gyroscope, or any combination thereof. In addition, for example, the first positioning circuit 520 can employ components such as an accelerometer, a gyroscope, or a global positioning system (GPS) receiver to identify the current location on the earth through satellite triangulation of cellular tower triangulation, including calculating pseudo-ranges from satellites or cellular towers.

[0128] The first positioning circuit 520 may include a first positioning interface 532. The first positioning interface 532 may be used for communication between the first positioning circuit 520 and other functional units or circuits in the first device 102 (including the optical sensor 108).

[0129] The first positioning interface 532 can receive information from other functional units / circuits or from external sources, or can send information to other functional units / circuits or to external destinations. External sources and external destinations refer to sources and destinations external to the first device 102, including satellites and cellular towers.

[0130] The first positioning interface 532 may include different implementations, depending on which functional units / circuits or external units / circuits are interfaced with the first positioning circuit 520. The first positioning interface 532 may be implemented using technologies and techniques similar to the implementation of the first control circuit 512, including analog circuits, digital circuits, wireless circuits, or wired circuits.

[0131] The second device 106 can be optimized to implement embodiments of the present invention in a multi-device embodiment with the first device 102. The second device 106 can provide additional or higher performance processing capabilities compared to the first device 102. The second device 106 can include a second control circuit 534, a second communication circuit 536, a second user interface 538, and a second storage circuit 546.

[0132] The second user interface 538 allows an operator (not shown) to interface and interact with the second device 106. The second user interface 538 may include an input device and an output device. Examples of input devices of the second user interface 538 may include a keypad, a touch pad, soft keys, a keyboard, a microphone, or any combination thereof, to provide data and communication input. Examples of output devices of the second user interface 538 may include a second display interface 540. The second display interface 540 may include a display, a projector, a video screen, a speaker, or any combination thereof.

[0133] The second control circuit 534 can execute the second software 542 to provide the intelligence of the second device 106 of the computing system 100. The second software 542 can operate in conjunction with the first software 526. Compared to the first control circuit 512, the second control circuit 534 can provide additional capabilities.

[0134] The second control circuit 534 can operate the second user interface 538 to display information. The second control circuit 534 can also execute second software 542 for other functions of the computing system 100, including operating the second communication circuit 536 to communicate with the first device 102 over the network 104.

[0135] The second control circuit 534 can be implemented in many different ways. For example, the second control circuit 534 can be a processor, an embedded processor, a microprocessor, hardware control logic, a hardware finite state machine (FSM), a digital signal processor (DSP), or a combination thereof.

[0136] The second control circuit 534 may include a second control interface 544. The second control interface 544 may be used for communication between the second control circuit 534 and other functional units or circuits in the second device 106. The second control interface 544 may also be used for communication outside the second device 106.

[0137] The second control interface 544 can receive information from the other functional units / circuitry or from external sources, or can send information to the other functional units / circuitry or to external destinations. External sources and external destinations refer to sources and destinations external to the second device 106 .

[0138] The second control interface 544 can be implemented in different ways and can include different implementations, depending on which functional units / circuits or external units / circuits are interfaced with the second control interface 544. For example, the second control interface 544 can be implemented using a pressure sensor, an inertial sensor, a micro-electromechanical system (MEMS), an optical circuit, a waveguide, a wireless circuit, a wired circuit, an analog circuit, or a combination thereof.

[0139] The second storage circuit 546 may store the second software 542. The second storage circuit 546 may also store information such as the feature lines 306 identified in the Nth frame 302 and the N+1th frame 304 captured by the first device 102. The second storage circuit 546 may be sized to provide additional storage capacity to supplement the first storage circuit 514.

[0140] For purposes of illustration, the second storage circuit 546 is shown as a single element, but it should be understood that the second storage circuit 546 can be a distribution of storage elements. In addition, for purposes of illustration, the computing system 100 is shown with the second storage circuit 546 as a single hierarchical storage system, but it should be understood that the computing system 100 can include a second storage circuit 546 in a different configuration. For example, the second storage circuit 546 can be formed using different storage technologies to form a memory hierarchy system that includes different levels of cache, main memory, rotating media, or offline storage.

[0141] The second storage circuit 546 may be a controller of a volatile memory, a nonvolatile memory, an internal memory, an external memory, or a combination thereof. For example, the second storage circuit 546 may be a controller of a nonvolatile storage device such as a nonvolatile random access memory (NVRAM), a flash memory, a disk storage device, or a volatile storage device such as a static random access memory (SRAM).

[0142] The second storage interface 548 can receive information from the other functional units / circuitry or from external sources, or can send information to the other functional units / circuitry or to external destinations. External sources and external destinations refer to sources and destinations external to the second device 106 .

[0143] The second storage interface 548 may include different implementations, depending on which functional units / circuits or external units / circuits are interfaced with the second storage circuit 546. The second storage interface 548 may be implemented using technologies and techniques similar to the implementations of the second control interface 544. The second storage circuit 546 may support the ZNCC tracking unit 114 to track the Nth frame 302 and the N+1th frame 304 captured by the first device 102. Figure 3 The characteristic line 306 is shown in FIG.

[0144] The second communication circuit 536 can enable external communications to and from the second device 106. For example, the second communication circuit 536 can allow the second device 106 to communicate with the first device 102 over the network 104. For example, the second device 106 can provide the threshold value derived from the ZNCC tracking unit 114 to other vehicles in or near the area of ​​interest 203 of the first device 102.

[0145] The second communication circuit 536 may also function as a communication hub, thereby allowing the second device 106 to function as part of the network 104 and is not limited to being an endpoint or terminal unit or circuit of the network 104. The second communication circuit 536 may include active and passive circuit components, such as microelectronics, wireless circuits, wired circuits, analog circuits, or antennas, for interacting with the network 104.

[0146] The second communication circuit 536 may include a second communication interface 550. The second communication interface 550 may be used for communication between the second communication circuit 536 and other functional units or circuits in the second device 106. The second communication interface 550 may receive information from other functional units / circuits, or may send information to other functional units or circuits.

[0147] The second communication interface 550 can include different implementations, depending on which functional units or circuits are interfaced with the second communication circuit 536. The second communication interface 550 can be implemented using technologies and techniques similar to the implementation of the second control interface 544, including circuits, waveguides, wireless circuits, wired circuits, analog circuits, or combinations thereof.

[0148] The second communication circuit 536 can be coupled to the network 104 to send information to the first device 102, including the optimized parameters 120 in the second device transmission 510. The first device 102 can receive information from the second device transmission 510 of the network 104 in the first communication circuit 516. The computing system 100 can be executed by the first control circuit 512, the second control circuit 534, or a combination thereof. For illustrative purposes, the second device 106 is shown as having a partition including a second user interface 538, a second storage circuit 546, a second control circuit 534, and a second communication circuit 536, but it should be understood that the second device 106 can include different partitions. For example, the second software 542 can be divided differently so that some or all of its functions can be located in the second control circuit 534 and the second communication circuit 536. In addition, for clarity, the second device 106 may include Figure 5 Other functional units or circuits not shown.

[0149] The functional units or circuits in the first device 102 can work individually and independently of other functional units or circuits. The first device 102 can work individually and independently of the second device 106 and the network 104.

[0150] The functional units or circuits in the second device 106 can work individually and independently of other functional units or circuits. The second device 106 can work individually and independently of the first device 102 and the network 104.

[0151] The above functional units or circuits may be implemented in hardware. For example, one or more functional units or circuits may be implemented using a gate array, an application specific integrated circuit (ASIC), a circuit, a processor, a computer, an integrated circuit, an integrated circuit core, a pressure sensor, an inertial sensor, a micro-electromechanical system (MEMS), a passive device, a physical non-transitory memory medium containing instructions for executing software functions, a portion thereof, or a combination thereof.

[0152] For illustrative purposes, the computing system 100 is described through the operation of the first device 102 and the second device 106. It should be understood that the first device 102 and the second device 106 can operate any modules and functions of the computing system 100.

[0153] For further example, the first device 102 may be an autonomous vehicle or a driver assistance vehicle. The first interface circuit 518 may be from Figure 1 The optical sensor 108 receives input for supporting the ZNCC tracking unit 114. The ZNCC tracking unit 114 can be used to identify and track the feature line 306 identified by the first control circuit 512 from the Nth frame 302 and the N+1th frame 304 captured by the optical sensor 108 and the first positioning circuit 520.

[0154] It has been discovered that the second device 106 can accommodate the fast line detection unit 116 from the first device 102, which provides fast detection of characteristic lines captured in the sensor data stream in the region of the region of interest 203. As an example, the second control circuit 534 can verify the optimized parameters 120 and pass the information to the second storage circuit 546 for analysis.

[0155] Reference now Figure 6 , which shows an embodiment of the present invention Figure 1 601 of an exemplary operation flow chart of a computing system 100. The exemplary operation flow chart 601 of the computing system 100 shows a module 602 collecting image frames, wherein the sensor data stream 111 is captured in the sensor data local storage device 110 for analysis and processing. The sensor data local storage device 110 can maintain several sequences of the Nth frame 302 and the N+1th frame 304 for analysis.

[0156] The process may proceed to the distortion removal module 604, where Figure 1 The distortion correction unit 112 can identify and correct the Figure 1 The optical sensor 108 Figure 2 The fisheye lens 221 leads to Figure 1 The optical distortion 113. The distortion correction unit 112 can analyze Figure 2 The overlap region 232 of the optical sensor 108 is analyzed to identify and correct the optical distortion 113. It should be understood that different versions of the optical sensor 108 may produce different amounts of optical distortion 113. By analyzing the overlap region 232, the amount and type of optical distortion 113 may be identified and corrected.

[0157] Flow then proceeds to an image resizing module 606, where the sensor data stream 111 may be filtered to reduce the size of the image file stored in the sensor data local storage 110. For example, the file may be converted from red, green, and blue (RGB) to grayscale, or a certain number of pixels captured by the optical sensor 108 may be filtered. The reduced size of the image captured in the Nth frame 302 and the N+1th frame 304 may allow for faster identification of the image and feature lines 306.

[0158] The process then proceeds to the fast line detection module 608, where the Nth frame 302 and the N+1th frame 304 can be processed to extract the feature line 306 in the image. The extraction of the feature line 306 can be performed by Figure 1 The fast line detection unit 116 is used to perform the transform and can be performed up to three times faster than the classic Hough transform when operating on the same image.

[0159] The flow then proceeds to module 610 of ZNCC tracking for matching feature lines 306 between the Nth frame 302 and the N+1th frame 304 . Figure 1 The ZNCC tracking unit 114 may calculate a ZNCC score 611 for a set of lines that may be a match. Figure 5 The first control circuit 512 can compare the ZNCC score 611 to the ZNCC threshold 612 established during the training of the ZNCC tracking unit 114. If the ZNCC score 611 is above the ZNCC threshold 612, then the two lines being compared are considered a match for the feature line 306. Figure 2 When the characteristic line 306 is identified in the region of interest 203, the first control circuit 512 can determine the optical sensor 108 Figure 1 , which allows the GPS coordinates 115 of the characteristic line 306 to be identified.

[0160] The process then proceeds to the published module 614, where the first control circuit 512 can remove any extraneous lines caused by reflections and noise. The remaining feature lines in the feature lines 306 can be verified and GPS coordinates 115 applied. The published feature lines 306 can be compared with the GPS coordinates from Figure 1 The data of the high-definition map 120 is compared to control the first device 102 in the autonomous driving vehicle mode.

[0161] It has been found that the combination of the fast line detection module 609 and the ZNCC tracking module 610 can provide high quality line extraction to identify the feature line 306 in terms of both clarity and position identification. By utilizing the information obtained in the Nth frame 302, the N+1th frame 304 can be quickly verified and the GPS coordinates 115 applied.

[0162] Reference now Figure 7 , which shows an exemplary line identification mechanism 701 in an embodiment. The exemplary line identification mechanism 701 shows an image 702, where Figure 1 The characteristic lines 306 extracted from the sensor data local storage device 110 have been Figure 1 The distortion correction unit 112 processes the image.

[0163] First, a line 704 is selected which is defined by two endpoints, namely a top point 706 and a bottom point 708. Several sampling points 710 are extracted from the line 704. The sampling points 710 should have the same distance from one to the next. The sampling points 710 may represent pixels 712 from the image 702.

[0164] Next, a pixel patch 714 is identified around each of the sample points 710 identified in the line 704. The pixel patch 714 may be an array of pixels 712 centered at the sample point 710, such as a 7×7 array. The pixel patch 714 centered at the sample point 710 is extracted. The size of the pixel patch 714 may be selected based on a machine learning algorithm to maximize the efficiency of line identification. The sample point 710 being analyzed may be located at the center 716 of the array. For example, the value of the pixel 712 may be a color grade, a density number, a grayscale value, etc.

[0165] Reference now Figure 8 , which shows an exemplary line identification schematic diagram 801 through neighboring lines in previous frames. The exemplary line identification schematic diagram 801 shows the Nth frame 302 and the N+1th frame 304 in the line identification process.

[0166] The first control circuit 512 can be based on Figure 7 The top point 706 and Figure 7 The first control circuit 512 may calculate the real world coordinates 802 of a midpoint 804 located at the center of the feature line 306 being analyzed, such as Figure 1 The first control circuit 512 may analyze the characteristic line 306 that is physically close to the target line 808. The direction of the line relative to the X-axis 810 may be represented by an angle θ806 between the target line 808 and the X-axis 810.

[0167] First control circuit 512 may identify neighboring line 812 in Nth frame 302 based on angle θ 806 and midpoint 804 of target line 808. Matching line 814 in Nth frame 302 may be identified as a match for target line 808 when the difference between angle θ 806 and real-world position 802 is less than a dedicated threshold.

[0168] For all adjacent lines 812 in the Nth frame 302, the first control circuit 512 may extract adjacent lines 812 having the same number and position as the target line 808. Figure 7 The first control circuit 512 can also extract the same size near the sampling point 710. Figure 7 Pixel patch 714.

[0169] The first control circuit 512 may calculate for all pixel patches 714 adjacent to the line 812 Figure 6 The ZNCC score is 611. If the score is Figure 6If the ZNCC score 611 of the sample point 710 is above the ZNCC threshold 612, it is considered a good match. The first control circuit 512 can calculate all ZNCC scores 611 of the sample point 710. The first control circuit 512 can identify the matching line 814 of the target line 808 as the feature line 306.

[0170] If the first control circuit 512 determines that more than one adjacent line 812 has the same value of ZNCC score 611, then the adjacent line 812 with the smaller difference in angle θ 806 or the smaller distance from the midpoint 804 will be selected as the matching line 814. This situation may occur when analyzing the reflection of one of the adjacent lines 812.

[0171] The first control circuit 512 can check if multiple matches are detected for the target line 808. When this process is complete, each target line 802 should have a single match line 814. If the first control circuit 512 detects that more than one match line 814 is mapped to a single target line 808, or more than one target line 808 is mapped to a single neighboring line 812, the first control circuit 512 will select the neighboring line 812 with the smaller difference in angle θ 806 and the smaller difference in distance of the intermediate point 804.

[0172] Reference now Fig. 9 , which shows an embodiment of the present invention Figure 1 Flowchart of a method 900 for operating the computing system 100 of the embodiment of the present invention. The method 900 includes: in box 902, capturing a sensor data stream of an area of ​​interest provided by an optical sensor; in box 904, correcting optical distortion of the optical sensor for the area of ​​interest; in box 906, extracting line segments from the Nth frame of the area of ​​interest; in box 908, calculating the zero-mean normalized cross-correlation ZNCC score of the line segments in the Nth frame and the N+1th frame; in box 910, identifying a feature line by having a ZNCC score greater than a ZNCC threshold; and in box 912, calculating the GPS coordinates of the feature line to control the device.

[0173] The resulting method, process, apparatus, device, product and / or system is straightforward, cost-effective, uncomplicated, highly versatile, accurate, sensitive and effective, and can be implemented by adapting known components for easy, efficient and economical manufacture, application and utilization. Another important aspect of the embodiments of the present invention is that it valuable supports and serves the historical trend of reducing costs, simplifying systems and improving performance.

[0174] Accordingly, these and other valuable aspects of embodiments of the present invention advance the state of the technology to at least the next level.

[0175] Although the present invention has been described in conjunction with a specific best mode, it should be understood that many substitutions, modifications and variations will be apparent to those skilled in the art based on the foregoing description. Therefore, it is intended to include all such substitutions, modifications and variations that fall within the scope of the included claims. All matters set forth herein or shown in the accompanying drawings should be interpreted in an illustrative and not restrictive sense.

Claims

1. A computing system comprising: A control circuit, the control circuit being configured to: storing the sensor data stream of the area of ​​interest provided by the optical sensor, correcting optical distortions of the optical sensor in the sensor data stream for the region of interest, Extract line segments from the Nth frame of the region of interest, Calculate the zero-mean normalized cross-correlation ZNCC score of the line segment in the Nth frame and the N+1th frame, The feature line is identified by the ZNCC score being greater than a ZNCC threshold, and The GPS coordinates of the characteristic line are calculated to control the device.

2. The computing system of claim 1, wherein: The control circuit is configured to calculate the ZNCC score including: the control circuit is configured to: applying a sampling point to the line segment; Extracting pixel patches centered at the sampling point in the Nth frame and the N+1th frame; and The similarity (EQ1) of the pixel patches in the Nth frame and the N+1th frame is calculated to identify which of the line segments are used for the ZNCC score.

3. The computing system of claim 1, wherein: The control circuit is configured to identify the optical distortion by analyzing overlapping areas of the regions of interest.

4. The computing system of claim 1, wherein: The control circuit is configured to extract the line segment in the following manner: Identify lines from frame N; Applying the midpoint at the GPS coordinates of the center of the line; Calculate the angle θ between the line and the X-axis of the Nth frame; as well as The GPS coordinates of the midpoint in the Nth frame and the N+1th frame and the angle θ of the line are compared to identify the extracted line segment as the feature line.

5. The computing system of claim 1, wherein: The control circuit is configured to calculate the GPS coordinates of the feature line including verifying the GPS coordinates of the feature line using a high definition map.

6. The computing system of claim 1, wherein: The control circuit is configured to generate an Nth frame by combining the sensor data streams from the front optical sensor, the rear optical sensor, the left optical sensor, and the right optical sensor and correcting the optical distortion.

7. The computing system of claim 1, wherein: The control circuit is configured to correct the optical distortion of the optical sensor including individually adjusting a front optical sensor, a rear optical sensor, a left optical sensor, and a right optical sensor when creating the Nth frame.

8. A method for operation of a computing system, comprising: storing a sensor data stream of a region of interest provided by an optical sensor; correcting optical distortion of the optical sensor in the sensor data stream for the region of interest; Extracting line segments from the Nth frame of the region of interest; Calculating a zero-mean normalized cross-correlation ZNCC score of the line segment in the Nth frame and the N+1th frame; identifying a feature line by the ZNCC score being greater than a ZNCC threshold; as well as The GPS coordinates of the characteristic line are calculated to control the device.

9. The method according to claim 8, wherein: Calculating the ZNCC score includes: applying a sampling point to the line segment; Extracting pixel patches centered at the sampling point in the Nth frame and the N+1th frame; and The similarity (EQ1) of the pixel patches in the Nth frame and the N+1th frame is calculated to identify which of the line segments are used for the ZNCC score.

10. The method of claim 8, further comprising analyzing overlapping regions of the regions of interest to identify the optical distortion.

11. The method according to claim 8, wherein the method comprises extracting the line segment by: Identify lines from frame N; Applying the midpoint at the GPS coordinates of the center of the line; calculating the angle θ between the line and the X-axis; and The GPS coordinates of the midpoint in the Nth frame and the N+1th frame and the angle θ of the line are compared to identify the extracted line segment as the feature line.

12. The method of claim 8, wherein calculating the GPS coordinates of the feature line comprises verifying the GPS coordinates of the feature line using a high definition map.

13. The method of claim 8, further comprising generating an Nth frame by combining the sensor data streams from a front optical sensor, a rear optical sensor, a left optical sensor, and a right optical sensor and correcting the optical distortion. 14 . The method of claim 8 , wherein correcting the optical distortion of the optical sensors comprises individually adjusting a front optical sensor, a rear optical sensor, a left optical sensor, and a right optical sensor when creating the Nth frame.

15. A non-transitory computer readable medium comprising instructions for a computing system, the instructions, when executed, causing the computing system to perform operations comprising: storing a sensor data stream of a region of interest provided by an optical sensor; correcting optical distortion of the optical sensor in the sensor data stream for the region of interest; Extracting line segments from the Nth frame of the region of interest; Calculating a zero-mean normalized cross-correlation ZNCC score of the line segment in the Nth frame and the N+1th frame; identifying a feature line by the ZNCC score being greater than a ZNCC threshold; as well as The GPS coordinates of the characteristic line are calculated to control the device.

16. The non-transitory computer readable medium comprising instructions of claim 15, wherein calculating the ZNCC score comprises: applying a sampling point to the line segment; Extracting pixel patches centered at the sampling point in the Nth frame and the N+1th frame; as well as The similarity (EQ1) of the pixel patches in the Nth frame and the N+1th frame is calculated to identify which of the line segments are used for the ZNCC score.

17. The non-transitory computer readable medium comprising instructions of claim 15, wherein the operations include analyzing overlapping regions of the regions of interest to identify the optical distortion.

18. The non-transitory computer readable medium comprising instructions according to claim 15, wherein the operations comprise extracting the line segments by: Identify lines from frame N; Applying the midpoint at the GPS coordinates of the center of the line; calculating the angle θ between the line and the X-axis; and The GPS coordinates of the midpoint in the Nth frame and the N+1th frame and the angle θ of the line are compared to identify the extracted line segment as the feature line.

19. The non-transitory computer-readable medium comprising instructions of claim 15, wherein the operations comprise generating an Nth frame by combining the sensor data streams from a front optical sensor, a rear optical sensor, a left optical sensor, and a right optical sensor and correcting the optical distortion.

20. The non-transitory computer readable medium comprising instructions of claim 15, wherein correcting the optical distortion of the optical sensor comprises individually adjusting a front optical sensor, a rear optical sensor, a left optical sensor, and a right optical sensor when creating the Nth frame.

Images