Data processing apparatus, data processing method, and data processing program

Abstract

To provide a data processing device, a data processing method, and a program capable of identifying a plurality of types of vehicles from an image.SOLUTION: The data-processing device includes an acquiring unit 292 that acquires a video of a moving image to be analyzed captured by a monitoring camera or the like, a vehicle-extraction processing unit 294 that performs processing of extracting vehicles by making an HTTP request to an external server including a pre-learned image-recognition model that performs image-recognition such as YOLO, and a vehicle-identification processing unit 296 that performs processing of identifying types of vehicles by generating a prompt for requesting identification of types of vehicles extracted by the vehicle-extraction processing unit 294 and inputting the prompt to a first generative AI model such as Gemini (Internet retrieval <URL:https: / / gemini.google.com / ?hl=ja>).SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a data processing device, a data processing method, and a data processing program. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Incidentally, when conducting road traffic volume surveys, people often sit on the side of the road, check and count the types of vehicles passing by, and compile the counting results to create a report, but this requires people to go to the side of the road.In recent years, therefore, efforts have been made to automate this process by using AI to identify vehicles from images captured by surveillance cameras and other devices.

[0005] However, when AI (Artificial Intelligence) identifies vehicles from video, it is sometimes necessary to classify them into multiple types (e.g., eight vehicle models). In this case, a large amount of data for each vehicle model is required, which poses a challenge for identification, making it difficult to identify specific vehicle models. [Means for solving the problem]

[0006] A first aspect of the technology of the present disclosure is a data processing device including an acquisition unit that acquires video of a vehicle, a vehicle extraction processing unit that extracts the vehicle and its vehicle registration plate from the video acquired by the acquisition unit using a pre-trained image recognition model, and a vehicle identification processing unit that identifies the type of vehicle from the extraction result of the vehicle extraction processing unit using a pre-trained first generation AI model.

[0007] A second aspect of the technology of the present disclosure is a data processing method in which a computer acquires video of a vehicle, extracts the vehicle and its registration plate from the acquired video using a pre-trained image recognition model, and performs a process of identifying the type of vehicle from the extraction result using a pre-trained first generation AI model.

[0008] A third aspect of the technology of the present disclosure is a data processing program for causing a computer to acquire video footage of a vehicle, extract the vehicle and its registration plate from the acquired video using a pre-trained image recognition model, and identify the type of vehicle from the extraction results using a pre-trained first generation AI model. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a conceptual diagram illustrating an example of a configuration of a data processing system. [Figure 2] FIG. 2 is a conceptual diagram showing an example of main functions of a data processing device and a user terminal. [Figure 3] 1 shows a part of the outline of the specific process. [Figure 4] 4 shows another part of the outline of the specific process. [Figure 5] 2 shows a schematic functional configuration of a specific processing unit of the data processing device. [Figure 6] 10 is a diagram illustrating an example of an operational flow of specific processing by a data processing device. [Figure 7]10 shows an example of a graph summarizing traffic volume information for each vehicle type. [Figure 8] An example of a map reflecting the results of a traffic volume survey is shown below. [Figure 9] 10 is a schematic diagram illustrating a functional configuration of a modified example of a specific processing unit of a data processing device. [Figure 10] FIG. 10 is a conceptual diagram showing a modified example of the main functions of the data processing device. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, an example of an embodiment of a data processing device, a data processing method, and a program according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0011] First, the terms used in the following description will be explained.

[0012] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), or an APU (Accelerated Processing Unit).

[0013] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0014] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0015] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0016] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0017] FIG. 1 shows an example of the configuration of a data processing system 10 according to the embodiment.

[0018] As shown in FIG. 1, the data processing system 10 includes a data processing device 12 and a user terminal 14. An example of the data processing device 12 is a server. An example of the user terminal 14 is a personal computer or a smartphone. In this embodiment, the data processing device 12 is an example of a "data processing device" according to the technology of the present disclosure.

[0019] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0020] The user terminal 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0021] The reception device 38 is equipped with a keyboard, a mouse, etc., and receives user input. The reception device 38 may also receive user input by touch of an indicator (for example, a pen or a finger) by detecting contact with the touch panel, or may receive user input by voice by detecting the user's voice by a microphone. The control unit 46A transmits data indicating the received user input to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0022] The output device 40 includes a display 40A and a speaker 40B, and presents data to a person by outputting the data in a form of expression that can be perceived by the person (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46.

[0023] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0024] FIG. 2 shows an example of the main functions of the data processing device 12 and the user terminal 14.

[0025] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0026] In the user terminal 14, the processor 46 performs the reception output processing. The storage 50 stores a reception output program 62. The reception output program 62 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 62 from the storage 50 and executes the read reception output program 62 on the RAM 48. The reception output processing is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 62 executed on the RAM 48.

[0027] Next, the processing of the specific processing unit 290 when the data processing device 12 performs specific processing to investigate road traffic volume from video will be described.

[0028] The identification processing unit 290 in this embodiment utilizes video footage from surveillance cameras such as CCTV (closed-circuit television) installed on roads, extracts vehicles from the video footage, identifies the vehicle types, and converts the data into data for recording. That is, as shown in FIG. 3, video footage captured by a surveillance camera or the like is input to the data processing device 12. In the data processing device 12, the identification processing unit 290 extracts vehicles and automobile registration plates (hereinafter referred to as license plates) from the input video footage using image recognition, and identifies the vehicle type from the extracted vehicle image and license plate. The example in FIG. 3 shows an example in which a vehicle with a Nagoya license plate is extracted and recognized as a commercial vehicle. The identification results are then converted into data for recording.

[0029] 4, the specific processing unit 290 performs processing to create a report of the traffic volume survey results using the digitized and recorded information. For example, a report summarizing traffic volume information for each vehicle type is created. Then, by performing format conversion or other processing on the created report to digitize it, the traffic volume survey results are superimposed and displayed on a map or the like.

[0030] In detail, the identification processing unit 290 has the functions of an acquisition unit 292, a vehicle extraction processing unit 294, a vehicle identification processing unit 296, a report creation processing unit 298, and a digitization processing unit 300, as shown in FIG.

[0031] The acquisition unit 292 acquires video of the moving image to be analyzed, which has been captured by a surveillance camera or the like, from the user terminal 14. Note that the acquisition unit 292 may acquire video of the moving image to be analyzed directly from a surveillance camera or the like.

[0032] The vehicle extraction processing unit 294 performs processing to extract vehicles present in the video. In this embodiment, the vehicle extraction processing unit 294 performs processing to extract vehicles by making an HTTP request to an external server equipped with an image recognition model that performs image recognition such as YOLO (You Only Look Once), which has been trained in advance.

[0033] The vehicle identification processing unit 296 performs processing to identify the type of vehicle extracted. In this embodiment, the processing to identify the type of vehicle is performed by making an HTTP request to an external server equipped with a first generation AI (Artificial Intelligence) model that has been trained in advance. For example, a prompt requesting identification of the type of vehicle extracted by the vehicle extraction processing unit 294 is generated, and the request is sent to Gemini (Internet search engine). <url: https: gemini.google.com ?hl="ja">) or the like, a process of identifying the type of vehicle is performed. The results of the identification of the type of vehicle are stored in a database 302 constructed in the storage 32 or the like. Specifically, in this embodiment, a prompt is generated to request identification of the type of vehicle as one of eight types (light passenger car, passenger car, bus, light freight vehicle, small freight vehicle, freight-passenger car, regular freight vehicle, and special vehicle) along with the image extracted by the vehicle extraction processing unit 294, and is input to Gemini, thereby identifying the type of vehicle.

[0034] The report creation processing unit 298 performs processing to create a report on the traffic volume survey results based on the information stored in the database 302. In this embodiment, the report creation processing unit 298 performs processing to create a report on the traffic volume survey results by using the information stored in the database 302 via an HTTP request to an external server equipped with a second generation AI model that has been trained in advance. For example, the report creation processing unit 298 uses the information stored in the database 302 to generate a prompt requesting the creation of a report on the traffic volume survey results, and sends the request to ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> The report is generated by inputting prompts to the second generation AI model such as the first generation AI model 11. The generated report is then output to the user terminal 14 for display.

[0035] The digitization processing unit 300 performs processes such as digitizing the report created by the report creation processing unit 298 by converting the format of the report and displaying the traffic volume survey results for each road on a map. In this embodiment, the digitization processing unit 300 digitizes the report via an HTTP request to an external server equipped with a third generation AI model and displays it on the map. Specifically, the digitization processing unit 300 generates a prompt requesting the server to convert the format of the created report, digitize it, and superimpose the traffic volume survey results for each road on the map. The prompt is then input to a third generation AI model such as GPTs (a customized, proprietary ChatGPT), which digitizes the report and generates a map on which the traffic volume survey results for each road are superimposed. This allows the map reflecting the survey results for each road to be viewed on a network 54 such as the Internet.

[0036] In this embodiment, the prompts are generated automatically in each processing unit, but the prompts may also be generated manually.

[0037] Each generative AI model receives a prompt containing an instruction, and also receives at least one of inference data such as voice data representing a voice, text data representing a text, and image data representing an image. Each generative AI model performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data, text data, or image data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0038] Next, the operation of the data processing system 10 will be described.

[0039] An example of the flow of the identification process will be described with reference to Fig. 6. The flow of the identification process shown in Fig. 6 is an example of a "data processing method" according to the technique of the present disclosure.

[0040] In step S100, the acquisition unit 292 acquires video of the moving image to be analyzed, and the process proceeds to step S102. For example, the user operates the user terminal 14 to transmit the image to be analyzed to the data processing device 12, and the acquisition unit 292 acquires the video to be analyzed from the user terminal 14.

[0041] In step S102, the vehicle extraction processing unit 294 performs processing to extract vehicles present in the video, and then proceeds to step S104. For example, an HTTP request is made to an external server equipped with an image recognition model that performs image recognition such as YOLO, to extract vehicles from the video to be analyzed, and the extraction results are obtained from the external server. The vehicle extraction results are then stored in the database 302.

[0042] In step S104, the vehicle identification processing unit 296 performs processing to identify the type of vehicle, and then proceeds to step S106. For example, a prompt requesting identification of the type of vehicle extracted by the vehicle extraction processing unit 294 is generated, and input to a generation AI such as Gemini to identify the type of vehicle and obtain the identification result. The vehicle identification result is stored in the database 302.

[0043] In step S106, the report creation processing unit 298 performs processing to create a report of the traffic volume survey results based on the information stored in the database 302, and then proceeds to step S108. For example, a prompt requesting the creation of a report of the traffic volume survey results is generated using the information stored in the database 302, and the report is created by inputting the prompt into a generation AI such as ChatGPT, and the creation results are obtained. For example, as shown in FIG. 7, a report summarizing traffic volume information for each vehicle type is created. The created report is then stored in the database 302. Note that FIG. 7 shows an example of a graph summarizing traffic volume information for each vehicle type.

[0044] In step S108, the digitization processing unit 300 performs digitization processing, and the process proceeds to step S110. The digitization processing involves digitizing the report created by the report creation processing unit 298 by converting its format and displaying the traffic volume survey results for each road superimposed on a map. For example, a prompt requesting that the created report be formatted and digitized and superimposed on a map is generated, and this is input to a generation AI such as GPTs to generate a map with the traffic volume survey results for each road superimposed. The generated map is then output to a network 54 such as the Internet, allowing users to view the map reflecting the survey results. For example, as shown in FIG. 8, the map may display, for example, "Traffic signals need to be adjusted at intersections where congestion occurs" as a congestion occurrence point, "At 8:00 a.m., there is a high volume of foot traffic for regular freight vehicles, and the degree of congestion is high," as a peak-hour traffic volume, or "Public transportation should be promoted due to the high volume of private vehicles passing through" as a traffic volume by use.

[0045] In step S110, the acquisition unit 292 determines whether or not there is a next video. If the determination is affirmative, the process returns to step S100 to repeat the above-described processing, and if the determination is negative, the series of identification processing ends.

[0046] By performing this identification process, it is possible to extract and identify vehicles from the images they appear in. Furthermore, the vehicle identification results can be used to easily create a traffic volume survey report.

[0047] In addition, since data identifying the type of vehicle can be provided immediately, it can contribute to improving the efficiency of planned road repairs.

[0048] It is also possible to contribute to the collection of road passability information. For example, as shown in FIG. 9, the identification processing unit 290 may further include a creation processing unit 304 that creates passability information based on the acquired video and the identification results of the vehicle identification processing unit 296. Alternatively, the identification processing unit 290 may include an acquisition unit 292, a vehicle extraction processing unit 294, a vehicle identification processing unit 296, and a creation processing unit 304. The creation processing unit 304 extracts from the video those vehicles that were passable among the vehicles identified by the vehicle identification processing unit 296 and identifies the vehicle type, thereby creating passability information for each vehicle type. This makes it possible to determine the passability of each vehicle type from road video and provide passability information immediately in the event of a disaster.

[0049] In the above embodiment, the vehicle extraction processing unit 294, the vehicle identification processing unit 296, the report creation processing unit 298, and the digitization processing unit 300 each perform their respective processes by making an HTTP request to an external server. However, this is not limiting. For example, as shown in FIG. 10 , the image recognition model 58A, the first generation AI model 58B, the second generation AI model 58C, and the third generation AI model 58D may be stored in the storage 32 of the data processing device 12 and used by the specific processing unit 290, respectively. Alternatively, at least one of the image recognition model 58A, the first generation AI model 58B, the second generation AI model 58C, and the third generation AI model 58D may be stored in the storage 32, and the other models may be requested to be processed by an external server via an HTTP request. Furthermore, although the first generation AI model 58B, the second generation AI model 58C, and the third generation AI model 58D have been described as separate models, they may be realized by a single generation AI model, or a model integrating at least two or more models may be used.

[0050] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[0051] In the above embodiment, an example was given in which a specific process is performed by one computer 22, but the technology disclosed herein is not limited to this, and distributed processing of the specific process may be performed by multiple computers including computer 22.

[0052] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[0053] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0054] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[0055] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[0056] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[0057] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[0058] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[0059] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[0060] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[0061] The following additional notes are provided regarding the above-described embodiments. <Appendix 1> an acquisition unit that acquires a video of a vehicle; a vehicle extraction processing unit that extracts a vehicle and its registration plate from the video acquired by the acquisition unit using a pre-trained image recognition model; a vehicle identification processing unit that identifies the type of vehicle from the extraction result of the vehicle extraction processing unit using a first generation AI model that has been trained in advance; 2. A data processing device comprising:

[0062] <Appendix 2> A data processing device as described in Appendix 1, further including a report creation processing unit that creates a road traffic volume report based on the vehicle type identification results using a pre-trained second generation AI model.

[0063] <Appendix 3> 3. The data processing device of claim 2, further comprising a digitization processing unit that digitizes the report using a pre-trained third generative AI model.

[0064] <Appendix 4> 4. The data processing device according to any one of appendices 1 to 3, further comprising a creation processing unit that creates road passability information based on the video and the identification result of the vehicle identification processing unit.

[0065] <Appendix 5> The computer Acquires footage of the vehicle, extracting a vehicle and its registration plate from the acquired image using a pre-trained image recognition model; A data processing method that uses a pre-trained first generation AI model to perform a process of identifying the vehicle type from the extraction results.

[0066] <Appendix 6> On the computer, Acquires footage of the vehicle, extracting a vehicle and its registration plate from the acquired image using a pre-trained image recognition model; A data processing program for executing a process to identify the vehicle type from the extraction results using a pre-trained first generation AI model. [Explanation of symbols]

[0067] 10 Data Processing System 12 Data Processing Device 14 User terminal 56 Specific Processing Program 56A Image Recognition Model 56B First Generation AI Model 56C Second Generation AI Model 56D Third Generation AI Model 290 Special Processing Department 292 Acquisition Department 294 Vehicle Extraction Processing Unit 296 Vehicle Identification Processing Unit 298 Report Creation Processing Unit 300 Digitalization Processing Unit 302 Database 304 Creation Processing Unit< / url:>

Claims

1. an acquisition unit that acquires a video of a vehicle; a vehicle extraction processing unit that extracts a vehicle and its registration plate from the video acquired by the acquisition unit using a pre-trained image recognition model; a vehicle identification processing unit that identifies the type of vehicle from the extraction result of the vehicle extraction processing unit using a first generation AI model that has been trained in advance; 2. A data processing device comprising:

2. The data processing device according to claim 1 , further comprising a report creation processing unit that creates a road traffic volume report based on the vehicle type identification result using a second generation AI model that has been trained in advance.

3. The data processing device according to claim 2 , further comprising a digitization processing unit that digitizes the report using a third generative AI model that has been trained in advance.

4. The data processing device according to claim 1 , further comprising a creation processing unit that creates road passability information based on the video and the identification result of the vehicle identification processing unit.

5. The computer Acquires footage of the vehicle, extracting a vehicle and its registration plate from the acquired image using a pre-trained image recognition model; A data processing method that uses a pre-trained first generation AI model to perform a process of identifying the type of vehicle from the extraction results.

6. On the computer, Acquires footage of the vehicle, extracting a vehicle and its registration plate from the acquired image using a pre-trained image recognition model; A data processing program for executing a process of identifying the type of vehicle from the extraction results using a first generation AI model that has been trained in advance.

Images