Systems and methods for hybrid cloud-edge processing

Abstract

A traffic sensor includes an image capturing unit that captures images of traffic and generates source video data therefrom. The traffic sensor includes an image processing unit that processes the source video data via image recognition to generate a recognition record comprising a dataset of one or more values representing image recognized vehicle and / or traffic characteristics. The traffic sensor includes a controller that determines whether the source video data corresponding to the recognition record requires enhanced image recognition processing. The traffic sensor includes a transceiver that transmits the recognition record with the corresponding source data to a server when enhanced image recognition processing is required.

Description

SYSTEMS AND METHODS FOR HYBRID CLOUD-EDGEPROCESSINGBACKGROUND

[0001] The present invention relates to traffic monitoring systems and methods, and more particularly to such systems and methods for distributing video data processing between edge processing (i.e., processing on traffic sensors) and cloud / server processing.

[0002] Roadway traffic is generally monitored for traffic characteristics, such as vehicle count, vehicle type (e.g. class, make, model, color, year, etc.), as well as movements and interrelationships (speeds, congestion, following distance, accidents, near misses, etc.). This data is then used by agencies (e.g., the Department of Transportation, Law Enforcement, etc.) to develop and implement policy, as well as perform enforcement, and dynamic adaptions to mitigate problems and undesirable conditions. Accordingly, there is a continuous need to expand and enhance the information content being extracted from the roadway through roadside sensors.

[0003] The algorithms required for this continually growing informational extraction need frequently require a processing load that exceeds the capability of the traffic sensors deployed in the field. This limits the performance capabilities of the system and / or requires excessive optimization cycles to enable new algorithms to “fit” on the traffic sensors.

[0004] As a potential solution, cloud processing has virtually unlimited processing capacity, and is therefore capable of supporting the ever-increasing processing loads. Moreover, cloud processing offers streamlined development cycles and allows for non-optimal efficiencies. However, the the cost / complexity to transfer continuous video data from the traffic sensors, which gather the data, to the cloud is very high.

[0005] Accordingly, for video data processing to occur in the cloud, it requires reduced informational content (e.g. down sampled video) and / or it imposes an excessive transfer burden. And for video data processing to occur on the edge, i.e., in the traffic sensors, it requires an increase in the edge processing capabilities and / or it imposes performance, development and optimization cycle limitations.

[0006] Traffic monitoring systems and methods are therefore needed that are capable of effectively and efficiently utilizing cost-effective but less powerful edge processing in conjunction with more powerful but less cost-effective server / cloud based processing.BRIEF SUMMARY OF THE INVENTION

[0007] Traffic monitoring systems and methods are disclosed for distributing video data processing between edge processing (i.e., processing on traffic sensors) and cloud / server processing.

[0008] Other objects, advantages and novel features of the present invention will become apparent from the following detailed description of one or more preferred embodiments when considered in conjunction with the accompanying drawings. It should be recognized that the one or more examples in the disclosure are non-limiting examples and that the present invention is intended to encompass variations and equivalents of these examples.BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The features, objects, and advantages of the present invention will become more apparent from the detailed description, set forth below, when taken in conjunction with the drawings, in which like reference characters identify elements correspondingly throughout.

[0010] Figure 1 illustrates an exemplary traffic monitoring system in accordance with at least one embodiment of the invention.

[0011] Figure 2 illustrates an exemplary method for traffic monitoring in accordance with at least one embodiment of the invention.DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] The above described drawing figures illustrate the present invention in at least one embodiment, which is further defined in detail in the following description. Those having ordinary skill in the art may be able to make alterations and modifications to what is described herein without departing from its spirit and scope. While the present invention is susceptible of embodiment in many different forms, there is shown in the drawings and will herein be described in detail at least one preferred embodiment of the invention with the understanding that the present disclosure is to be considered as an exemplification of the principles of thepresent invention, and is not intended to limit the broad aspects of the present invention to any embodiment illustrated.

[0013] In accordance with the practices of persons skilled in the art, the invention is described below with reference to operations that are performed by a computer system or a like electronic system. Such operations are sometimes referred to as being computer-executed. It will be appreciated that operations that are symbolically represented include the manipulation by a processor, such as a central processing unit, of electrical signals representing data bits and the maintenance of data bits at memory locations, such as in system memory, as well as other processing of signals. The memory locations where data bits are maintained are physical locations that have particular electrical, magnetic, optical, or organic properties corresponding to the data bits.

[0014] When implemented in software, code segments perform certain tasks described herein. The code segments can be stored in a processor readable medium. Examples of the processor readable mediums include an electronic circuit, a semiconductor memory device, a read-only memory (ROM), a flash memory or other non-volatile memory, a floppy diskette, a CD-ROM, an optical disk, a hard disk, etc.

[0015] In the following detailed description and corresponding figures, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, it should be appreciated that the invention may be practiced without such specific details. Additionally, well-known methods, procedures, components, and circuits have not been described in detail.

[0016] The present invention generally relates to traffic monitoring systems and methods, and more particularly to such systems and methods for distributing video data processing between edge processing (i.e., processing on traffic sensors) and cloud / server processing.

[0017] Figure 1 is a schematic representation of a traffic monitoring system 100 in accordance with one or more aspects of the invention.

[0018] As shown in Figure 1, the traffic monitoring system 100 comprises one or more traffic sensors 200 communicatively coupled to a system server 300, via a network 800. In general, the traffic monitoring system 100 enables the collection of traffic related data for transmission to a third-party server 400, which may be a law-enforcement server, a civilianservice server, a department of traffic (or other road / traffic monitoring entities) server, and / or some other commercial data server, via the network 800.

[0019] Each traffic sensor 200 comprises an imaging device 210, a controller 220, a video indexing unit 230, an image processing unit 240, a memory 250, and a transceiver 260, each communicatively coupled to a common data bus 270 that enables data communication between the respective components.

[0020] The imaging device 210 captures images of traffic, in particular, video images of vehicles 110 making up the traffic, and generates video data therefrom. The imaging device 210 may be a video camera of any camera type, which captures video images suitable for computerized image recognition of objects within the captured images. For example, the camera may utilize charge-coupled-device (CCD), complementary metal-oxide- semiconductor (CMOS) and / or other imaging technology, to capture standard, night- vision, infrared, and / or other types of images, having predetermined resolution, contrast, color depth, and / or other image characteristics. The video data may be timestamped so as to indicate the date and time of recording. It will be understood that while video images and video data are referred to herein, the principles of the invention are equally applicable to non-video images and nonvideo image data.

[0021] The controller 220 controls the operation of the other components of the traffic sensor 200 in accordance with the functionalities described herein. The controller may be one or more processors programmed to carry out the described functionalities in accordance with software stored in the memory. The controller may be a standard processor, such as a central processing unit (CPU), graphics processing unit (GPU), or a dedicated processor, such as an application-specific integrated circuit (ASIC) or field programmable gate array (FPGA), or portion thereof.

[0022] The video indexing unit 230 is configured to index the source video data and save it to the memory. In operation, the video indexing unit 230 tags the source video data with one or more identifiers by which the source video data may be identified. Such identifiers include, for example, an index value, a timestamp of the source video, a geolocation of where the source video data was recorded, and / or a sensor ID of the traffic sensor recording the source video data.

[0023] The image processing unit 240 applies computerized image processing techniques to the video data (i.e., source video data) captured by the imaging device 210, so as to generate a recognition record that is transmitted to the system server 300. Some exemplary types of image processing that may be applied to the source video data include image enhancement, encoding, compression, down-sampling, and recognition processing.

[0024] The recognition record is a dataset preferably comprising one or more values reflecting image recognized vehicle and / or traffic characteristics. These characteristic values may be associated with corresponding confidence scores indicating the confidence with which the particular characteristic value was determined.

[0025] Accordingly, in some embodiments, the image processing unit 240 may apply computerized image recognition techniques to identify objects within the video images. For example, the image processing unit 240 may identify individual vehicles 110 captured by the video images, as well as their associated characteristics. These vehicle characteristics may include, for example, vehicle type, class, make, model, color, year, drive type (e.g., electric, hybrid, etc.), license plate number, registration, trajectory, speed, location, etc., or any combination thereof.

[0026] Moreover, in some embodiments, the image processing unit 240 may apply image analysis techniques to the image-recognized video images so as to identify captured traffic characteristics. These traffic characteristics may include, for example, temporal histories, vehicle counts, congestion levels, the presence of accidents, disabled vehicles, foreign objects, or other traffic incidents, or any combination thereof.

[0027] The recognition record may also comprise processed video data. The processed video data may be low-resolution or low-bit video data and / or limited frame video data. That is, the processed video data may be of lower resolution / bit and / or more limited in frames than the source video data from which it is generated. Thus, the processed video data is reduced in data size when compared to the corresponding source video data. In some embodiments, the recognition record may comprise the processed video data having metadata that includes the dataset with one or more of the characteristic values discussed herein.

[0028] In addition, the recognition record may include the identifier (e.g., timestamp, geolocation data, sensor ID, etc.) associated with the corresponding source video data fromwhich it was generated. Accordingly, the identifier may be used to identify the source video data corresponding to the recognition record.

[0029] In some embodiments, the controller 220 is further configured to indicate to the video indexing unit 230 that the recognition record requires enhanced computerized image recognition that is more accurate than the computerized image recognition that the image processing unit 210 applies. Such indication may be based on a determination by the controller 220 that image recognition beyond what is provided for by the traffic sensor 200 may be beneficial to a business objective of the traffic monitoring service.

[0030] For example, the indication may be in response to determining that one or more confidence scores of the recognition record are below predetermined thresholds for corresponding characteristic values. This may correspond to a situation where there are potential inaccuracies that should be resolved by enhanced computerized image recognition available at the server.

[0031] Additionally, or alternatively, the indication may be in response to determining that the characteristic values of recognition record match a subset of corresponding values for vehicles-of-interest and / or particular traffic conditions. This may correspond to a situation where the detailed analysis of classes of vehicles and / or traffic conditions is desired.

[0032] In response to such indication, the video indexing unit 230 may identify the source video data by its identifier, which corresponds to the identifier of the recognition record indicated as requiring enhanced computerized image recognition. The source video may thereafter be transmitted to the system server 300 for enhanced computerized image recognition in association with the recognition record.

[0033] The memory 250 stores software and data that can be accessed by the processor(s), and includes both transient and persistent storage. The transient storage is configured to temporarily store data being processed or otherwise acted on by other components, and may include a data cache, RAM or other transient storage types. The persistent storage is configured to store software and data until actively deleted. In at least some embodiments, the memory 250 may store any data used by the traffic sensor 200 in furtherance of the functions described herein.

[0034] The transceiver 250 communicatively couples the traffic sensor 200 to the network 800 so as to enable data transmission therewith. The network 800 may be any type of network, wired or wireless, configured to facilitate the communication and transmission of data, instructions, etc., and may include a local area network (LAN) (e.g., Ethernet or other IEEE 802.03 LAN technologies), Wi-Fi (e.g., IEEE 802.11 standards, wide area network (WAN), virtual private network (VPN), global area network (GAN)), a cellular network, or any other type of network or combination thereof.

[0035] The system server 300 is generally configured to provide centralized support for the traffic sensors 200. The system server 300 is configured to receive, store and / or process traffic sensor generated data, from each of the traffic sensors 200. In particular, the system server 300 is a server of the traffic monitoring service.

[0036] The system server 300 is further generally configured to receive data from the one or more traffic sensors 200, and to store such data in one or more databases (not shown). The system server 300 may request data from each traffic sensor, individually or collectively, in accordance with a set schedule and / or on an ad hoc basis.

[0037] In at least one embodiment, the system server 300 comprises an analysis engine (not shown), which is configured to analyze historical data stored in the database to generate vehicle and / or traffic metrics for various periods of time. The metrics may be generated via statistical analysis of the historical data, or by comparison of the historical data with secondary data sets (e.g., manufacturer identified weight, emissions, etc. of make / model), or any combination thereof. The metrics may be, for example, vehicle tonnage, emissions, drive types, number, etc., over a section of the roadway per period of time.

[0038] The system server 300 may store the metrics in the database for later retrieval, update, modification, deletion, etc. In at least one embodiment, the system server 300 transmits the metrics, via the network 800, to the third-party server 400.

[0039] The third-party server 400 is generally configured to send and receive data to / from the system server 300. The third-party server 400 may be one or more servers of law- enforcement (e.g., police, highway patrol, sheriff, etc.), civil service (e.g., department of transportation, municipality, etc.), and private (e.g., trucking company, security, etc.) entities.

[0040] In general, each server many include one or more server computers connected to the network 800. Each server computer may include computer components, including one or more processors, memories, displays and interfaces, and may also include software instructions and data for executing the functions of the server described herein. The servers may also include one or more storage devices configured to store large quantities of data and / or information, and may further include one or more databases. For example, the storage device may be a collection of storage components, or a mixed collection of storage components, such as ROM, RAM, hard-drives, solid-state drives, removable drives, network storage, virtual memory, cache, registers, etc., configured so that the server computers may access it.

[0041] In at least one embodiment, the analysis engine of the system server 300 is further configured to apply enhanced computerized image recognition techniques to identify objects within the source video data. The enhanced computerized image recognition techniques applied by the system server 300 differ from the computerized image recognition techniques applied by the traffic sensor 200 in that the enhanced computerized image recognition techniques identify vehicle / traffic characteristic values more accurately, i.e., with higher confidence scores. Accordingly, the enhanced image processing techniques may be carried out via an artificial neural network and / or cloud processing, which provide more computing power than the traffic sensors.

[0042] In some embodiments, the recognition record corresponding to the source video data on which the enhanced computerized image recognition techniques is applied is updated to include the vehicle / traffic characteristic values determined via the enhanced computerized image recognition techniques. Such updating may include replacing the original values or appending the new values to the recognition record. By the same token, the recognition record may be updated to include the source video data in addition to or in place of the processed video data. Accordingly, an enhanced recognition record is generated and may be stored in the server databases for use by the server, including the analysis engine.

[0043] Figure 2 is a flow-chart representing an exemplary method 20 of operation for the traffic monitoring system in accordance with one or more aspects of the invention.

[0044] At Step 21, respective traffic sensors 200 capture images of vehicle traffic, namely, video images of passing vehicles, and generate the source video data therefrom. The traffic sensors 200 are preferably each positioned at various roadway locations where the vehicletraffic is to be monitored. The imaging devices 210 are preferably positioned such that the captured images include the respective license plates of the passing vehicles, as well as other vehicle characteristics, e.g., vehicle type, class, make, model, color, year, drive type, license plate number, registration, trajectory, speed, location, etc., or any combination thereof.

[0045] At Step 22, the video indexing unit 230 tags the source video with one or more identifiers by which the source video may be identified, and saves the source video to the memory 250.

[0046] At Step 23, the image processing unit 240 applies computerized image processing techniques to the source video data so as to generate the recognition record, which is transmitted to the system server 300.

[0047] At Step 24, the controller 220 indicates to the video indexing unit 230 that the recognition record requires enhanced computerized image recognition.

[0048] At Step 25, the system server 300 performs enhanced computerized image recognition on the source video data in association with the indicated recognition record.

[0049] In this manner, the traffic monitoring system 100 effectively and efficiently utilizes cost-effective but less powerful edge processing in conjunction with more powerful but less cost-effective server / cloud-based processing.

[0050] Additional aspects of the invention are disclosed in the Appendix attached hereto, the entire contents of which are incorporated herein by reference.

[0051] The embodiments described in detail above are considered novel over the prior art and are considered critical to the operation of at least one aspect of the described systems, methods and / or apparatuses, and to the achievement of the above described objectives. The words used in this specification to describe the instant embodiments are to be understood not only in the sense of their commonly defined meanings, but to include by special definition in this specification: structure, material or acts beyond the scope of the commonly defined meanings. Thus, if an element can be understood in the context of this specification as including more than one meaning, then its use must be understood as being generic to all possible meanings supported by the specification and by the word or words describing the element.

[0052] The definitions of the words or drawing elements described herein are meant to include not only the combination of elements which are literally set forth, but all equivalent structure, material or acts for performing substantially the same function in substantially the same way to obtain substantially the same result. In this sense, it is therefore contemplated that an equivalent substitution of two or more elements may be made for any one of the elements described and its various embodiments or that a single element may be substituted for two or more elements.

[0053] Changes from the subject matter as viewed by a person with ordinary skill in the art, now known or later devised, are expressly contemplated as being equivalents within the scope intended and its various embodiments. Therefore, obvious substitutions now or later known to one with ordinary skill in the art are defined to be within the scope of the defined elements. This disclosure is thus meant to be understood to include what is specifically illustrated and described above, what is conceptually equivalent, what can be obviously substituted, and also what incorporates the essential ideas.

[0054] Furthermore, the functionalities described herein may be implemented via hardware, software, firmware or any combination thereof, unless expressly indicated otherwise. If implemented in software, the functionalities may be stored in a memory as one or more instructions on a computer readable medium, including any available media accessible by a computer that can be used to store desired program code in the form of instructions, data structures or the like. Thus, certain aspects may comprise a computer program product for performing the operations presented herein, such computer program product comprising a computer readable medium having instructions stored thereon, the instructions being executable by one or more processors to perform the operations described herein. It will be appreciated that software or instructions may also be transmitted over a transmission medium as is known in the art. Further, modules and / or other appropriate means for performing the operations described herein may be utilized in implementing the functionalities described herein.

[0055] The foregoing disclosure has been set forth merely to illustrate the invention and is not intended to be limiting. Since modifications of the disclosed embodiments incorporating the spirit and substance of the invention may occur to persons skilled in the art, the invention should be construed to include everything within the scope of the description and equivalents thereof.

Claims

CLAIMS1. A traffic sensor, comprising: an image capturing unit configured to capture images of traffic and generate source video data therefrom; an image processing unit configured to process the source video data via image recognition to generate a recognition record comprising a dataset of one or more values representing image recognized vehicle and / or traffic characteristics; a controller configured to determine whether the source video data corresponding to the recognition record requires enhanced image recognition processing; and a transceiver configured to transmit the recognition record with the corresponding source data to a server when enhanced image recognition processes is required.

2. The traffic sensor of claim 1, wherein the transceiver is further configured to transmit the recognition record to the server without the corresponding source video data when enhanced image recognition processing is not required.

3. The traffic sensor of claim 1, wherein the enhanced image recognition processing is more accurate than the image recognition processing of the image processing unit.

4. The traffic sensor of claim 1, wherein the one or more values is each associated with a confidence score, and wherein enhanced image recognition processing is required when the confidence score of at least one of the one or more values is below a predetermined threshold.

5. The traffic sensor of claim 1, wherein the recognition record includes processed video data corresponding to the source video data reduced in data size.

6. A traffic monitoring system, comprising: a traffic sensor, comprising: an image capturing unit configured to capture images of traffic and generate source video data therefrom,an image processing unit configured to process the source video data via image recognition to generate a recognition record comprising a dataset of one or more values representing image recognized vehicle and / or traffic characteristics, a controller configured to determine whether the source video data corresponding to the recognition record requires enhanced image recognition processing, and a transceiver configured to transmit the recognition record with the corresponding source data to a server when enhanced image recognition processes is required; and the server configured to: receive the recognition record and the corresponding source video data, process the corresponding source video data via the enhanced image recognition processing, and update the one or more values of the recognition record in accordance with the enhanced image recognition processing.

7. The traffic monitoring system of claim 6, wherein the transceiver is further configured to transmit the recognition record to the server without the corresponding source video data when enhanced image recognition processing is not required.

8. The traffic monitoring system of claim 6, wherein the enhanced image recognition processing is more accurate than the image recognition processing of the image processing unit.

9. The traffic monitoring system of claim 6, wherein the one or more values is each associated with a confidence score, and wherein enhanced image recognition processing is required when the confidence score of at least one of the one or more values is below a predetermined threshold.

10. The traffic monitoring system of claim 6, wherein the recognition record includes processed video data corresponding to the source video data reduced in data size.

11. A traffic monitoring method, comprising:generating, by an image capturing unit of a traffic sensor, source video data from images of traffic captured by the image capturing unit; processing, by an image processing unit of the traffic sensor, the source video data via image recognition to generate a recognition record comprising a dataset of one or more values representing image recognized vehicle and / or traffic characteristics; determining, by a controller of the traffic sensor, whether the source video data corresponding to the recognition record requires enhanced image recognition processing; and transmitting, by a transceiver of the traffic sensor, the recognition record with the corresponding source data to a server when enhanced image recognition processes is required.

12. The method of claim 11, wherein the recognition record is transmitted to the server without the corresponding source video data when enhanced image recognition processing is not required.

13. The method of claim 11, wherein the enhanced image recognition processing is more accurate than the image recognition processing of the image processing unit.

14. The method of claim 11, wherein the one or more values is each associated with a confidence score, and wherein enhanced image recognition processing is required when the confidence score of at least one of the one or more values is below a predetermined threshold.

15. The method of claim 11, wherein the recognition record includes processed video data corresponding to the source video data reduced in data size.

16. The method of claim 11, further comprising: receiving, by the server, the recognition record and the corresponding source video data, processing, by the server, the corresponding source video data via the enhanced image recognition processing, and updating, by the server, the one or more values of the recognition record in accordance with the enhanced image recognition processing.

Images