control device

Abstract

Improve the technology for calculating image importance. [Solution] The control device 20 identifies an object that is estimated to be captured by the in-vehicle camera 24A based on the location information of the vehicle 10, creates a control statement consisting of all or part of a description of the identified object, a predetermined instruction statement, and user preference information input via the input unit 25, and includes a control unit 27 that calculates the importance of one or more images included in the video captured by the in-vehicle camera 24A based on the created control statement.

Description

Technical Field

[0001] This disclosure relates to a control device.

Background Art

[0002] Conventionally, a technique for calculating the importance of an image has been known. For example, Patent Document 1 discloses a technique for assigning a predetermined importance to each landmark.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In Patent Document 1, since the importance of a landmark is not set according to the user's preference, when video information is extracted, there is a problem that video information inappropriate for the user is extracted. Therefore, there is room for improvement in the technique for calculating the importance of an image.

[0005] An object of the present disclosure made in view of such circumstances is to improve the technique for calculating the importance of an image.

Means for Solving the Problems

[0006] A control device according to an embodiment of the present disclosure is a control device including an in-vehicle camera, an input unit for inputting user preference information, and a control unit. The control unit identifies an object estimated to be imaged by the in-vehicle camera based on vehicle position information, and creates a control sentence consisting of all or part of a description of the identified object, a predetermined instruction sentence, and the user preference information. Based on the created control sentence, the importance of each of one or more images included in the video imaged by the in-vehicle camera is calculated. [Effects of the Invention]

[0007] According to one embodiment of this disclosure, the technique for calculating the importance of an image is improved. [Brief explanation of the drawing]

[0008] [Figure 1] This is a block diagram showing a schematic configuration example of a system according to one embodiment of the present disclosure. [Figure 2] This flowchart shows an example of the control device's operation. [Figure 3] This diagram illustrates one example of a method for identifying an object. [Modes for carrying out the invention]

[0009] (Summary of the embodiment) Referring to Figure 1, an overview of System 1 according to the embodiment of this disclosure will be described. System 1 comprises a vehicle 10, a control device 20 mounted on the vehicle 10, and a server device 30. The control device 20 and the server device 30 are communicably connected to a network 2, including, for example, the Internet and a mobile communication network.

[0010] Vehicle 10 is, for example, an automobile, but is not limited to that and may be any vehicle. An automobile is, but is not limited to, a gasoline car, a BEV (Battery Electric Vehicle), a HEV (Hybrid Electric Vehicle), a PHEV (Plug-in Hybrid Electric Vehicle), or an FCEV (Fuel Cell Electric Vehicle).

[0011] The control device 20 is a computer mounted on the vehicle 10. The control device 20 can communicate with the server device 30 via the network 2.

[0012] The server device 30 is a computer equipped with a database 32A that stores at least a description and location information of objects located in a predetermined area. The server device 30 can communicate with the control device 20 via the network 2.

[0013] (Control device configuration) As shown in Figure 1, the control device 20 includes a communication unit 21, a storage unit 22, a positioning unit 23, an imaging unit 24, an input unit 25, an output unit 26, and a control unit 27.

[0014] The communication unit 21 is equipped with a communication interface for wireless connection to the network 2. The communication interface for connecting to the network 2 supports, but is not limited to, mobile communication standards such as 4G (4th Generation) or 5G (5th Generation).

[0015] The storage unit 22 includes one or more memories. The memories are, for example, semiconductor memories, magnetic memories, or optical memories, but are not limited to these. Each memory included in the storage unit 22 may function as, for example, a main memory, an auxiliary memory, or a cache memory. The storage unit 22 stores any information used for the operation of the control device 20. For example, the storage unit 22 may store system programs, application programs, and embedded software. The storage unit 22 may also include an image storage area (image buffer) 22A for temporarily storing images captured by the in-vehicle camera 24A, which will be described later.

[0016] The positioning unit 23 is equipped with a GPS sensor 23A. The GPS sensor 23A measures the location information and time information of the vehicle 10.

[0017] The imaging unit 24 is equipped with an in-vehicle camera 24A. The in-vehicle camera 24A is a video camera that captures images (moving or still images) of the area outside the vehicle, in front of the vehicle 10 in the direction of travel.

[0018] The input unit 25 is configured to include at least one input interface capable of receiving an input by a user. The input interface may be an input screen provided integrally with the display screen 26A described later. Further, the input interface may be a microphone that picks up the user's speech.

[0019] The output unit 26 includes a display screen 26A. The display screen 26A is a display device such as, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescent) display. The display screen 26A displays the video imaged by the in-vehicle camera 24A.

[0020] The control unit 27 includes one or more processors, one or more programmable circuits, one or more dedicated circuits, or a combination thereof. The processor is, for example, a general-purpose processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), or a dedicated processor specialized for specific processing, but is not limited thereto. The programmable circuit is, for example, an FPGA (Field-Programmable Gate Array), but is not limited thereto. The dedicated circuit is, for example, an ASIC (Application Specific Integrated Circuit), but is not limited thereto. The control unit 27 controls the operation of the entire control device 20. As shown in FIG. 1, in the present embodiment, the control unit 27 includes a specifying unit 271, a creating unit 272, a calculating unit 273, and an extracting unit 274.

[0021] The specifying unit 271 refers to a database 32A provided in the server device 30, which includes at least a description text of the object and position information, and specifies an object estimated to be imaged by the in-vehicle camera 24A based on the position information of the vehicle 10.

[0022] The creation unit 272 creates a control sentence consisting of part or all of the description text of the object specified by the specifying unit 271, a predetermined instruction sentence, and the user's preference information, using a natural language model or a template.

[0023] Based on the control sentence created by the creation unit 272, the calculation unit 273 uses an image recognition model to recognize whether each of one or more images included in the video captured by the in-vehicle camera 24A contains an object, and calculates the importance of the image based on the information of the image to which the recognized object belongs.

[0024] The extraction unit 254 extracts highlights of the video according to the importance of each of the one or more calculated images.

[0025] (Configuration of the server device) As shown in FIG. 1, the server device 30 includes a communication unit 31, a storage unit 32, and a control unit 33.

[0026] The communication unit 31 includes a communication interface for wirelessly connecting to the network 2. The communication interface connecting to the network 2 corresponds to, for example, a mobile communication standard such as 4G or 5G or a wireless LAN (Local Area Network) standard, but is not limited thereto.

[0027] The storage unit 32 stores any information used for the operation of the server device 30, such as a system program, an application program, and embedded software. Further, the storage unit 32 includes a database 32A. The database 32A stores at least the description text and location information of the objects located in a predetermined area.

[0028] The control unit 33 includes one or more processors, one or more programmable circuits, one or more dedicated circuits, or a combination thereof, and controls the overall operation of the server device 30.

[0029] (Operation flow of the control device) The operation of the control device 20 according to this embodiment will be described with reference to Figure 2.

[0030] S101: The control unit 27 acquires the position information of the vehicle 10.

[0031] The position information of vehicle 10 is the position coordinates (Xv, Yv) of vehicle 10 measured by GPS sensor 13A. The measured position coordinates of vehicle 10 are output to control unit 27.

[0032] S102: The control unit 27 (specification unit 271) identifies an object that is estimated to be imaged by the onboard camera 24A based on the position information of the vehicle 10.

[0033] The object refers to landmarks such as towers and skyscrapers, which serve as directional references in a region, as well as natural objects such as trees and mountains. In this embodiment, the object is assumed to be "Tokyo Tower," but is not limited to this.

[0034] Figure 3 illustrates an example of a method for identifying an object. As shown in Figure 3(a), the on-board camera 24A can capture an area of ​​the grayed-out field of view in the direction of travel of the vehicle 10, starting from the vehicle's position coordinates (Xv, Yv). Objects a, b, and c are located in the direction of travel of the vehicle 10. Of these objects, it is presumed that object a will be captured by the on-board camera 24A.

[0035] Figure 3(b) shows information about objects located around a given coordinate (Xv, Yv) stored in the database 32A provided in the server device 30. The object information includes the object's latitude and longitude, and a description. The control unit 27 refers to Figure 3(b) stored in the database 32A and identifies object a that is estimated to be captured by the onboard camera 24A. The description of object a (text_a) is, for example, "Tokyo Tower". However, the method for identifying the object is not limited to the methods shown in Figures 3(a) and (b), and any method may be used.

[0036] S103: The control unit 27 (creation unit 272) creates a control statement S consisting of all or part of the following: a description of the identified object a, a predetermined instruction, and user preference information.

[0037] (i) The control unit 27 may use the description of object a as the control statement. For example, if the description of object a is "Tokyo Tower", then "Tokyo Tower" may be used as the control statement S1.

[0038] (ii) The control unit 27 may use a control statement that combines the description of object a with a predetermined instruction statement. For example, if the description is "Tokyo Tower" and the predetermined instruction statement is "Please answer whether it is included in the image", then the control statement S2 may be "Please answer whether Tokyo Tower is included in the image".

[0039] (iii) The control unit 27 may use a control statement that combines the description of object a, a predetermined instruction, and user preference information input by the user via the input unit 25. For example, if the description of object a is "Tokyo Tower", the predetermined instruction is "Please answer whether it is included in the image. Also, please answer whether the image conforms to the user preference information expressed in the following sentence," and the user preference information is "I like Tokyo's skyscrapers", then the control statement S3 may be "Please answer whether Tokyo Tower is included in the image. Also, please answer whether the image conforms to the user preference information expressed in the following sentence: 'I like Tokyo's skyscrapers'."

[0040] S104: The control unit 27 acquires the video image Mp captured by the onboard camera 24A.

[0041] The control unit 27 may temporarily store the captured video Mp in the video storage area 22A.

[0042] S105: The control unit 27 (calculation unit 273) calculates the importance of each of the one or more images contained in the video Mp captured by the in-vehicle camera 24A based on the created control message.

[0043] The captured video Mp contains one or more images. The one or more images are still images. The control unit 27 adopts, for example, control statement S3 from among the control statements S exemplified in S103. Control statement S3 is: "Please answer whether Tokyo Tower is included in the image. Also, please answer whether the image is in line with the user's preference information expressed in the following sentence: 'I like Tokyo's skyscrapers'."

[0044] The control unit 27 uses a predetermined image recognition model to determine whether each of the one or more images in the video Mp captured by the in-vehicle camera 24A contains "Tokyo Tower." As a specific example of how to determine whether each of the one or more images contains "Tokyo Tower," the control unit 27 may use the Open-Vocabulary object detection technology, which is well known in the field of image recognition. If the control unit 27 determines that one of the one or more images contains "Tokyo Tower," it may calculate the importance of the image by dividing the number of pixels containing "Tokyo Tower" by the total number of pixels in image p. For example, if the total number of pixels n1 in image p is 12 million pixels, and the number of pixels n2 in the part where "Tokyo Tower" is captured is 3 million pixels, the control unit 27 calculates the importance of image p as 0.25 by dividing n2 by n1. However, the method for calculating the importance of an image is not limited to this, and any method may be adopted.

[0045] S106: The control unit 27 determines whether the calculation of the importance of all images, one or more of the images included in the video Mp captured by the in-vehicle camera 24A, has been completed. If the calculation of the importance of all images has been completed, the process proceeds to S107; otherwise, it returns to S105.

[0046] S107: The control unit 27 (extraction unit 274) extracts highlights from the video based on the importance of all images, one or more of the images contained in the video MP.

[0047] The control unit 27 extracts video highlights that include images whose importance is calculated to be equal to or greater than the threshold α in S105. If the threshold α is set to, for example, 0.1, the control unit 27 may extract video highlights that are a combination of one or more video scenes that include images whose importance is calculated to be 0.1 or greater. A video scene is a part of a video MP that depicts a certain situation and consists of one or more images (still images).

[0048] As described above, the control device 20 according to this embodiment identifies an object that is estimated to be captured by the in-vehicle camera 24A based on the location information of the vehicle 10, creates a control statement consisting of all or part of a description of the identified object, a predetermined instruction statement, and user preference information input via the input unit 25, and calculates the importance of one or more images included in the video captured by the in-vehicle camera 24A based on the created control statement.

[0049] With this configuration, it becomes possible to calculate the importance of an image according to the user's preferences based on control statements consisting of all or part of the descriptive text of the object, predetermined instruction text, and user preference information included in the video captured by the in-vehicle camera 24A. Therefore, the technology for calculating image importance is improved in that the probability of extracting video highlights containing appropriate images for the user is increased.

[0050] While this disclosure has been described based on the drawings and embodiments, it should be noted that those skilled in the art may make various modifications and alterations based on this disclosure. Therefore, it should be noted that these modifications and alterations are within the scope of this disclosure. For example, the functions, etc., included in each component or step can be rearranged in a logically consistent manner, and multiple components or steps can be combined into one or divided into two.

[0051] For example, in the above-described embodiment, it is also possible to have an embodiment in which the configuration and operation of the control device 20 are distributed among multiple computers that can communicate with each other. Furthermore, for example, it is also possible to have an embodiment in which some or all of the components of the control device 20 are provided in the vehicle 10. For example, a navigation device mounted in the vehicle 10 may include some or all of the components of the control device 20.

[0052] Furthermore, it is also possible to implement an embodiment in which a general-purpose computer functions as the control device 20 according to the above embodiment. Specifically, a program describing the processing content that realizes each function of the control device 20 according to the above embodiment is stored in the memory of the general-purpose computer, and the processor reads and executes the program. Therefore, this disclosure can also be implemented as a program that can be executed by a processor, or as a non-temporary computer-readable medium that stores said program. [Explanation of Symbols]

[0053] 1 System 2 Network 10 vehicles 20 Control device 21,31 Communications Department 22,32 Storage part 22A Video memory area (video buffer) 23 Positioning Unit 23A GPS sensor 24 Imaging Department 24A In-Car Camera 25 Input section 26 Output section 26A Display screen (display) 27,33 Control Unit 30 Server devices 32A Database 271 Specific part 272 Creation Department 273 Calculation Section 274 Extraction part

Claims

[Claim 1] In-car camera and An input section for entering user preference information, A control device having a control unit, The control unit identifies an object that is estimated to be captured by the in-vehicle camera based on the vehicle's location information, creates a control statement consisting of all or part of a description of the identified object, a predetermined instruction statement, and user preference information, and calculates the importance of one or more images included in the video captured by the in-vehicle camera based on the created control statement.

Images