Image generation device, training device, video analysis system, image generation method, and program

Abstract

A training device 111 uses a video analysis usage condition to acquire usage-condition related information related to the usage condition from servers 102, 103 of a user environment 101 and Web sites 106, 107, 108 of a Web environment 105. Then, the training device 111 extracts item information, which is information corresponding to each image description item, from the usage-condition related information, and generates image description data representing an image expected to undergo video analysis if the extracted item information is used to perform video analysis. Then, the training device 111 uses the image description data to generate an image for training a model to be used for video analysis.

Description

Image generation device, learning device, video analysis system, image generation method and program

[0001] The present invention relates to machine learning.

[0002] In recent years, the application of technology for analyzing surveillance camera footage has progressed. For example, technology for estimating a person's entire body posture has been proposed, and applications are progressing in the fields of customer safety in stores and urban surveillance. Machine learning, particularly learning models such as neural networks, are widely used in video analysis technology. Patent Document 1 describes a method for fine-tuning a trained neural network model using test data (video and labels) acquired by a user device.

[0003] International Publication No. 2018 / 142766

[0004] Neural networks for video analysis may not be able to achieve sufficient analysis accuracy if the input video trends differ between when they are trained (developed) and when they are used by a user (operated). Examples of input video trends include those related to the shooting environment, such as lighting conditions, and statistical trends in the attributes of the analysis target, such as people in their 50s (e.g., many of them are in their 50s). The method using test data, as described in Patent Document 1, requires additional learning using video captured at the user's installation location for a certain period of time. In other words, there is a problem in that sufficient analysis accuracy cannot be achieved during the test data creation period.

[0005] In view of the above problems, the present invention aims to enable tuning of a camera to begin earlier than a method in which additional learning is performed based on test data obtained after the camera is installed in a specific location.

[0006] The image generation device of the present invention is characterized by having an acquisition means for acquiring related information related to the usage conditions of the video analysis based on the usage conditions of the video analysis, a description generation means for generating, based on the related information, description data representing images that are expected to be the subject of the video analysis when the video analysis is used, and an image generation means for generating, based on the description data, images for training a model to be used in the video analysis.

[0007] According to the present invention, tuning of the camera can be started earlier than with a method in which additional learning is performed based on test data obtained after the camera is installed in a specific location.

[0008] 1 is a diagram illustrating an example of the overall configuration of a video analysis system; FIG. 2 is a diagram illustrating an example of the hardware configuration of a learning device; FIG. 3 is a diagram illustrating an example of the functional configuration of a learning device; FIG. 4 is a flowchart illustrating processing executed by the learning device; FIG. 5 is a flowchart illustrating details of image description data generation processing; FIG. 6 is a diagram illustrating an example of order information; FIG. 7 is a diagram illustrating an example of a table in which sub-items are defined; FIG. 8 is a diagram illustrating an example of the functional configuration of a learning device; FIG. 9 is a flowchart illustrating details of image description data correction processing; FIG. 10 is a diagram illustrating an example of a feedback screen; and FIG. 11 is a diagram illustrating an example of an image used for learning.

[0009] Hereinafter, each embodiment will be described with reference to the accompanying drawings.

[0010] <First Embodiment> Fig. 1 shows an example of a video analysis system according to this embodiment. The video may be a video or a still image. The following mainly describes an example in which the video is a video. Video analysis is a process in which a subject to be analyzed is extracted from information captured in the video, and the movement and position of the subject are identified, thereby outputting an analysis result according to the video analysis function. The following mainly describes an example in which the subject is a person. In this embodiment, video analysis is performed by inputting the video into a video analysis model.

[0011] 1 , the video analysis system includes various devices provided in a user environment 101 of a user who uses a video analysis device 104, and a learning device 111 provided in a vendor environment 110 of a developer who develops a video analysis model. The various devices provided in the user environment 101 and the learning device 111 are connected to the Internet 112. Various websites provided in a web environment 105 and a search server 109 are connected to the Internet 112. The learning device 111 transmits and receives data to and from the various devices connected to the Internet 112.

[0012] The video analysis device 104 is an integrated device that combines a camera (e.g., a surveillance camera) that captures video and a device that inputs the captured video into a video analysis model to perform video analysis. The video analysis device 104 also has functions according to the purpose of the video analysis. The learning device 111 optimizes the video analysis model to match these functions. In this embodiment, the video analysis device 104 has a fall detection function, and during operation, the fall detection function is used to detect a person's fall from the captured video (input video). Note that the video analysis device 104 may also have a purchase analysis function, an intrusion detection function, a crowd detection function, a suspicious behavior detection function, etc., instead of or in addition to the fall detection function. In this embodiment, the video analysis device 104 is used in a store, but it may also be used outdoors, such as in buildings other than stores or in urban areas.

[0013] In this embodiment, the learning device 111 acquires information related to the usage conditions (usage condition related information) from the usage conditions of the video analysis device 104, and generates images representing input video expected when the video analysis device 104 is in operation based on the acquired usage condition related information. The learning device 111 then uses the generated images as training images for a video analysis model. In other words, training images similar to those that will be used when the video analysis device 104 is in operation can be prepared before the video analysis device 104 begins operation. The learning device 111 is an example of an image generation device.

[0014] The user environment 101 includes an internal mail server 102, a sales data management server 103, a video analysis device 104, and a PC 115 used by the user. The internal mail server 102 and the sales data management server 103 store various data related to the user's business and operations. The PC (personal computer) 115 is a terminal device used by the user. The PC 115 displays a UI screen received from the learning device 111 on a display device (not shown) and transmits information entered on the UI screen via an input device (not shown) to the learning device 111.

[0015] The Web environment 105 includes a user site 106 established by the user himself, a news site 107 established by a news organization, and an information site 108 that compiles various information.

[0016] The search server 109 searches various websites provided in the web environment 105 using the query (keywords) received from the learning device 111 and transmits the search results (URLs, etc.) to the learning device 111 .

[0017] 2 is a block diagram showing the hardware configuration of the learning device 111. The learning device 111 has a CPU 201, a ROM 202, a RAM 203, a secondary storage device 204, an input device 205, a display device 206, and a network I / F 207. These components are connected via a bus 208 to input and output data to and from each other.

[0018] The CPU (Central Processing Unit) 201 controls the entire learning device 111. A GPU (Graphics Processing Unit) may be used instead of or together with the CPU. The ROM 202 is a non-volatile memory that stores various programs and data. The RAM 203 is a volatile memory that stores frame image data and temporary data generated in the flowcharts described below. The secondary storage device 204 is a rewritable storage device such as a hard disk drive or flash memory that stores image information, programs, various setting information, and the like.

[0019] The input device 205 is a keyboard, mouse, or the like, and is used to input operations from the developer. The display device 206 is a liquid crystal display, or the like, and displays processing results, etc., to the developer. The network I / F 207 is a modem, LAN, or the like that connects to networks such as the Internet 112 or an intranet. The CPU 201 reads programs stored in the ROM 202, secondary storage device 204, etc. into the RAM 203 and executes them, thereby realizing processing corresponding to each step in the flowchart described below.

[0020] 3 is a block diagram showing the functional configuration of the learning device 111 according to this embodiment. The CPU 201 reads out a program stored in the ROM 202, the secondary storage device 204, etc., into the RAM 203 and executes it, thereby realizing each function shown in FIG.

[0021] The usage condition input unit 301 inputs the usage conditions of the video analysis device 104 used by the user.

[0022] The related information acquisition unit 302 collects information related to the use conditions input in the use condition input unit 301 (use condition related information) from the user environment 101 and the Web environment 105 via the network I / F 207 .

[0023] The related information storage unit 303 stores the use condition related information collected by the related information acquisition unit 302 in the secondary storage device 204, the RAM 203, or the like.

[0024] The image description generating unit 304 generates image description data based on the use condition related information stored in the related information storage unit 303 .

[0025] The image description storage unit 305 stores the image description data generated by the image description generation unit 304 in the secondary storage device 204, the RAM 203, or the like.

[0026] The image generation unit 306 generates an image and a correct label to be assigned to the image based on the image description data stored in the image description storage unit 305 .

[0027] The generated image storage unit 307 stores the generated image and the correct label generated by the image generation unit 306 in the secondary storage device 204, the RAM 203, or the like.

[0028] The trained model storage unit 308 stores, in the secondary storage device 204 or the like, a trained video analysis model corresponding to a function (here, the fall detection function) provided by the video analysis device 104. Note that a known machine learning method such as a neural network is used as the video analysis model.

[0029] The additional learning unit 309 performs additional learning of the video analysis model stored in the trained model storage unit 308 using the generated images and correct labels stored in the generated image storage unit 307.

[0030] The model providing unit 310 transmits the video analysis model obtained by additional learning to the video analysis device 104 via the network I / F 207 .

[0031] 4 is a flowchart showing the details of the processing executed by the learning device 111 according to this embodiment. In the following description, each process (step) is denoted by adding an S to the beginning, and the process (step) is omitted.

[0032] First, in S401, the usage condition input unit 301 acquires order information for the video analysis device 104. Then, the CPU 201 extracts the usage conditions of the video analysis device 104 to be used by the user from the acquired order information, and acquires the extracted usage conditions.

[0033] FIG. 6 shows an example of order information when the video analysis device 104 is used in a store. The order information is generated when a developer receives an order for the video analysis device 104 from a user, and is stored, for example, in a data server installed in the vendor environment 110. The usage conditions input unit 301 acquires the order information from the data server via the network I / F 207. The order information in FIG. 6 includes information corresponding to various items such as an order recipient 601, a function 602, a business type 603, a store using the device 604, a camera installation location 605, a camera used 606, camera settings such as the set resolution 607, operating hours 608, a price 609, and a delivery date 610. The order recipient 601 indicates the name of the store using the device. The function 602 indicates the video analysis function corresponding to the intended use of the video analysis device 104. The business type 603 indicates the type of store used. The store using the device 604 indicates the name of the store branch used. The camera used 606 indicates the model information of the camera used to capture the video.

[0034] In the following description, it is assumed that content corresponding to predetermined items related to the use conditions (items 601 to 610 in the example of FIG. 6) from among the items of the order information is extracted as the use conditions. Note that content corresponding to only some of the items 601 to 610 may be extracted. Furthermore, instead of being limited to the method of extracting the use conditions from the order information, the use condition input unit 301 may transmit a UI screen (not shown) for inputting the use conditions to the PC 115, and obtain information input by the user via the UI screen as the use conditions.

[0035] In S402, the related information acquisition unit 302 collects use condition related information based on the use conditions acquired in S401. The use condition related information includes the conditions of the image that is expected to be captured when shooting under the use conditions acquired in S401, and information for identifying those conditions. The image description storage unit 305 stores the collected use condition related information in the secondary storage device 204, RAM 203, etc. as needed.

[0036] The usage condition related information relates to the background of the image. For example, it relates to background objects, lighting conditions, shooting range, and image quality. Background objects are the types of objects expected to exist in the background, for example, product shelves in a store, or roads in a city. Furthermore, lighting conditions are information specifying sunlight conditions (weather and time of day) if outdoors, and the type of light source and window size if indoors. The influence of external light can be identified from the window size. The shooting range is, for example, lens information of the camera used (wide-angle or telephoto, focal length, etc.) and the installation position and angle of the camera. Image quality is the image quality of the entire image, for example, the performance and set resolution of the camera used. Note that there are no particular limitations on the information about the background of the image.

[0037] The usage condition information also relates to the attributes of people in the image, such as the characteristics of people who are expected to appear in the video (such as their age, sex, race, clothing, and posture).

[0038] The related information acquisition unit 802 searches the web environment 105 or the user environment 101 for information regarding the background of the image and the attributes of the people in the image, and acquires the information of the web page obtained as a result of the search as usage condition related information.

[0039] When acquiring the use condition related information from the Web environment 105 , the related information acquisition unit 802 acquires the use condition related information from the Web environment 105 using the search server 109 .

[0040] The related information acquisition unit 802 performs a search using, for example, the function 602 "fall detection" as a query. As a result of the search, the related information acquisition unit 802 acquires, for example, article information about fall fraud from the news site 107. Assume that this article information includes text information such as "Fall fraud occurred. A white man in his 50s wearing a dress shirt fell in a beverage counter and sued the store."

[0041] Furthermore, the related information acquisition unit 802 performs a search using, for example, "A super" of the order recipient 601 as a query. As a result of the search, the related information acquisition unit 802 acquires, for example, basic store information such as the sales floor layout and business hours from the user site 106 of A super.

[0042] The related information acquisition unit 802 also performs a search using, for example, the "Las Vegas store" of the store used 604 as a query. As a result of the search, the related information acquisition unit 802 acquires, for example, the racial and age demographic composition of Las Vegas from the information site 108.

[0043] The related information acquisition unit 802 also performs a search using, for example, the model name "M30" of the camera 606 in use as a query. As a result of the search, the related information acquisition unit 802 acquires, for example, a review article about the M30 from the information site 108. Assume that this review article contains text information about image quality, such as "High image quality during the day, but high noise at night."

[0044] When obtaining usage condition related information from the user environment 101, the related information acquisition unit 802 obtains the usage condition related information from the user environment 101 using a search server (not shown) that can access the in-house mail server 102 and the sales data management server 103.

[0045] The related information acquisition unit 802 acquires testimony information from other stores from the in-house mail server 102, such as "A man in his 50s at the New York store complained that he fell in the vegetable section and was injured."

[0046] Furthermore, the related information acquisition unit 802 acquires data on person attributes, such as "50% of the purchasing demographics are in their 40s, and 40% are in their 50s," from the sales data management server 103 .

[0047] In S403, the image description generation unit 304 generates image description data based on the usage condition related information acquired in S402. The image description data is data for generating images that are expected to be captured when using the video analysis device 104, and is described in text. In this embodiment, the image description data is generated according to a format defined for each video analysis function.

[0048] Below are examples of image description data templates for each video analysis function: 1. Fall detection function Generates an image with {image quality} of a person of {race} {age} {gender} wearing {clothing} falling at {time} in {filming location} 2. Gradient analysis function Generates an image with {image quality} of a person of {race} {age} {gender} wearing {clothing} reaching out at {time} in {filming location} 3. Intrusion detection function Generates an image with {image quality} of a person of {race} {age} {gender} wearing {clothing} entering {area} in {filming location} at {time}

[0049] The race, age, etc. in the brackets {} above represent image description items. The image description generation unit 304 extracts information corresponding to each image description item from the usage condition related information acquired in S402, and generates multiple variations of image description data using the extracted information and a template corresponding to the video analysis function. Hereinafter, information corresponding to the image description items may be referred to as item information. Note that the image description items shown in 1 to 3 above for the image description data are merely examples, and are not particularly limited as long as they relate to the background of the image and the attributes of the subject (here, a person) to be analyzed.

[0050] Note that the image description data examples 1 to 3 above describe positive cases where a detection target is present, but depending on the function, image description data for negative cases where the detection target is not present may also be necessary. For example, in the case of the fall detection function 1, image description data for negative cases can be generated by changing the description "falling" to something else such as "walking" or "standing."

[0051] The process of generating image description data executed in S403 will be described in detail below with reference to the flowchart of FIG.

[0052] In S501, the image description generation unit 304 acquires information corresponding to each image description item from the usage condition-related information acquired in S402. Specifically, the image description generation unit 304 removes unnecessary information, such as HTML tags, from web pages stored as usage condition-related information in the secondary storage device 204, RAM 203, or the like, and extracts text information. Next, if the extracted text information is in Japanese, it is broken down into parts of speech using morphological analysis, and information corresponding to each image description item is extracted by matching it with a dictionary that further classifies nouns into categories such as race, age, and gender. Alternatively, information about each image description item may be extracted using a large-scale language model such as Generative Pre-trained Transformer (GPT).

[0053] For example, from the aforementioned article information "A fall fraud occurred. A white man in his 50s wearing a dress shirt fell over himself in the drinks section and sued the store," the following items of information are acquired regarding the person's attributes: age "50," clothing "dress shirt," race "white," and gender "male." In addition, the location of the image, "drinks section," is acquired as item information regarding the background of the image.

[0054] Furthermore, from the review article mentioned above, "High image quality during the day, but high noise at night," the time "daytime" and image quality "high image quality," as well as the time "nighttime" and image quality "high noise" are acquired as item information related to the background of the image. As a result, the image description generating unit 304 generates the following image description data: "Generate a high-image quality image of a Caucasian man in his 50s wearing a dress shirt falling in a drinks section in the daytime" "Generate a high-noise image of a Caucasian man in his 50s wearing a dress shirt falling in a drinks section at night"

[0055] Next, in S502, the image description generation unit 304 refines the information corresponding to each image description item. Since the information corresponding to the image description items is extracted from the usage condition related information, it represents content that is close to the image that will actually be shot by the user. Therefore, it is desirable to ensure as many variations of images as possible that are generated from the information corresponding to the image description items. Therefore, the image description generation unit 304 increases the variation of image description data by refining the information (item information) corresponding to each image description item.

[0056] Specifically, the image description generation unit 304 adds information corresponding to sub-items to the item information. Sub-items are subordinate items for further detailing the item information. For example, if "drinks counter" is extracted as the "photography location," the image description generation unit 304 acquires the "area" within the counter as a sub-item and adds "in front of the shelf," "aisle," etc. corresponding to the "area." Furthermore, for example, if "shirt" is extracted as the "clothing," the image description generation unit 304 acquires "color" and "pattern" as sub-items of the shirt, and adds "white," "black," "red," etc. corresponding to the "color," and adds "solid," "striped," "checked," etc. corresponding to the "pattern." FIG. 7 shows an example of a table defining sub-items. The image description generation unit 304 acquires sub-items, which are subordinate items of the item information, and information corresponding to each sub-item by referencing a table such as that shown in FIG. 7.

[0057] For example, the image description generation unit 304 generates the following variations of image description data by detailing the shooting location "drinks counter" and the clothing item "shirt" for the above-mentioned image description data: "Generate a high-quality image of a Caucasian man in his 50s wearing a plain white shirt, falling in front of a drink shelf in the daytime" "Generate a high-quality image of a Caucasian man in his 50s wearing a plain black shirt, falling in front of a drink shelf in the daytime" "Generate a high-quality image of a Caucasian man in his 50s wearing a plain red shirt, falling in front of a drink shelf in the daytime" "Generate a high-quality image of a Caucasian man in his 50s wearing a white striped shirt, falling in front of a drink shelf in the daytime" (the rest is omitted).

[0058] As described above, by combining sub-items with information corresponding to each image description item and acquiring information corresponding to the sub-items, it is possible to increase the variety of image description data.

[0059] Here, the use condition related information includes information that is likely to have actually been photographed (highly credible), such as the testimonial information from other stores. The image description generation unit 304 may adjust the information obtained from such use condition related information with high importance so that more variations are generated. The importance here is determined, for example, depending on the source from which the use condition related information is obtained. The image description generation unit 304 may refine only image description items obtained from testimonial information, or may increase the number of sub-items for image description items obtained from testimonial information compared to image description items obtained from sources other than testimonial information. In this way, the image description generation unit 304 adjusts the number of image description data generated based on the use condition related information depending on the importance of the use condition related information.

[0060] Furthermore, for example, information on percentages such as "50% in their 40s, 40% in their 50s" may be extracted from the usage condition-related information acquired from the sales data management server 103 or the information site 108. In such a case, it is assumed that the people expected to appear in the video will have a similar percentage. Therefore, the image description generation unit 304 may adjust the percentages of image description data for ages "40s" and "50s" to 50% and 40%, respectively, of the total image description data to be generated.

[0061] Next, in S503, the image description generation unit 304 performs adjustments according to the image description items. Because the usage condition-related information represents typical content of the video that is actually shot, there is a bias, such as the inclusion of a large amount of specific information for certain image description items. However, some image description items may make it easier to detect people with specific attributes, such as race or gender, which can lead to ethical issues and other inconveniences. To address this, the image description generation unit 304 may perform adjustments so that the proportion of image description data in which a specific image description item is specific information does not exceed a threshold value. Specifically, if the proportion exceeds the threshold value, the generated image description data is randomly thinned out until it falls below the threshold value.

[0062] The image description generating unit 304 stores the plurality of image description data generated as described above in the secondary storage device 204, RAM 203, etc. In this way, the series of processes for generating image description data is completed.

[0063] Returning to the description of FIG. 4, in S404, the image generation unit 306 generates training images based on the image description data generated in S403. Specifically, it uses Contrastive Language-Image Pre-training (CLIP), a pre-trained model that allows text and images to be represented using common vectors. The image generation unit 306 encodes the image description data using CLIP as a text encoder, and uses a diffusion model or the like as conditioning when generating images. This makes it possible to obtain images that conform to the image description data. Since there may be multiple images that are similar to the encoded image description data, multiple images can be obtained from a single piece of image description data.

[0064] Next, in S405, the image generation unit 306 generates correct labels for the learning images generated in S404 using image description data. This correct label is defined for each template of the image description data. For the image description data item 1 above, "falling," the correct label is defined as "falling," and for the negative examples, "walking" and "standing," the correct label is defined as "not falling." The generated image storage unit 307 assigns correct labels to the multiple generated images generated in S404 and stores them in the secondary storage device 204, RAM 203, or the like.

[0065] Next, in S406, the additional learning unit 309 reads out a trained video analysis model from the secondary storage device 204 or the like. Video analysis models that have been trained in advance using existing images held by the developer for each video analysis function are stored in the secondary storage device 204. The additional learning unit 309 reads out the video analysis model that corresponds to the function to be used (in this embodiment, fall detection).

[0066] In S407, the additional learning unit 309 performs additional learning of the video analysis model read out in S406 based on the generated image and correct label stored in S405. Specifically, the CPU 201 may use the weights of the trained model as initial values, fix the weights of the front-stage part (backbone) that extracts basic features of the model, and update only the weights of the backbone part. Of course, the method is not limited to the above, and any learning method that adapts a model to a generated image, such as updating the weights of the entire model, may be used.

[0067] In S408, the model providing unit 310 determines whether the user has started installing the video analysis device 104. Specifically, the model providing unit 310 checks this based on a notification from the user, such as an email. The process waits in S408 until the model providing unit 310 determines that installation of the video analysis device 104 has started. If it is determined that installation of the video analysis device 104 has started, the process proceeds to S409.

[0068] In S409, the model providing unit 310 transmits the updated model that underwent additional learning in S407 to the video analysis device 104. For example, the video analysis device 104 may download the updated model from the learning device 111 via the Internet 112. The video analysis device 104 inputs the video captured by the camera into the video analysis model provided by the learning device 111 and performs video analysis. After that, the series of processes in the flowchart shown in FIG. 4 end.

[0069] According to the present embodiment, image description data representing video that is expected to be actually captured can be generated from information related to the user's usage conditions for the video analysis device, and learning images can be generated based on the image description data. This makes it possible to train a video analysis model using images similar to those seen when the user device is in operation, without requiring the user to provide video actually captured with the device. In other words, highly accurate video analysis can be performed from the start of operation of the user device.

[0070] <Embodiment 2> In embodiment 1, a method was described in which training images generated based on image description data are used directly for additional training. In embodiment 2, a method is described in which training images generated based on image description data are checked by a user, and feedback from the user is reflected before the images are used for additional training. This makes it possible to generate training images that are more suited to the usage environment. Note that a description of the parts common to embodiment 1 will be omitted, and the description will focus on the differences from embodiment 1.

[0071] Figure 8 is a block diagram showing the functional configuration of the learning device 111 according to this embodiment. A usage condition input unit 801, a related information acquisition unit 802, a related information storage unit 803, and an image description generation unit 804 in Figure 8 correspond to the usage condition input unit 301, the related information acquisition unit 302, the related information storage unit 303, and the image description generation unit 304 in Figure 3. Furthermore, an image description storage unit 805, an image generation unit 806, and a generated image storage unit 807 in Figure 8 correspond to the image description storage unit 305, the image generation unit 306, and the generated image storage unit 307 in Figure 3. Furthermore, a trained model storage unit 808, an additional training unit 809, and a model providing unit 810 in Figure 8 correspond to the trained model storage unit 308, the additional training unit 309, and the model providing unit 310 in Figure 3.

[0072] A generated image providing unit 811 transmits the image generated by the image generating unit 806 to the user's PC 115 via the network I / F 207 .

[0073] The feedback receiving unit 812 receives text information that is feedback input by the user on the PC 115 .

[0074] The image description correcting unit 813 corrects the image description data stored in the image description storage unit 805 .

[0075] The basic processing flow executed by the learning device 111 according to this embodiment is the same as the flowchart shown in Fig. 4. Hereinafter, the processing flow for making corrections based on user feedback, which is a characteristic process of this embodiment, will be described with reference to Fig. 9.

[0076] 4, in S901, the generated image providing unit 811 selects a generated image to provide to the user. Since it is not realistic to have the user check all generated images, the generated images are selected and provided. Note that the method of selecting images may be any method, such as randomly selecting a predetermined number of images, or selecting only images generated based on image description data in which predetermined image description items are specific information.

[0077] Next, in S902, generated image providing unit 811 transmits the generated image selected in S901 to PC 115 via the Internet 112. Note that generated image providing unit 811 does not provide the generated image as is, but transmits HTML format data for displaying a confirmation UI screen having a screen configuration such as that shown in Fig. 10 on PC 115 in order to receive feedback from the user. In this way, generated image providing unit 811 presents to the user a confirmation UI screen displaying the generated image selected in S901.

[0078] 10 shows an example of the screen configuration of a confirmation UI screen presented to the user. The confirmation UI screen is displayed on the display device of the PC 115. Images 1002 and 1003 (two images in this example) selected in S901 are displayed on the confirmation UI screen 1001. Images 1002 and 1003 are images generated from image description data. To the right of images 1002 and 1003 are input fields 1004 and 1005 where the user can enter text comments. When the user enters feedback in input fields 1004 and 1005 via an input device on the PC 115 and then presses send button 1006, the PC 115 transmits the contents (text information) of input fields 1004 and 1005 to the learning device 111 via the Internet 112.

[0079] Next, in S903, the feedback receiving unit 812 determines whether or not feedback (text information) has been received from the PC 115. The feedback receiving unit 812 waits in S903 until it receives feedback, and if it determines that it has received feedback, the processing proceeds to S904.

[0080] In S904, the image description correction unit 813 corrects the image description data stored in the secondary storage device 204, RAM 203, etc., based on the text information received in S903. The image description correction unit 813 acquires information corresponding to each image description item from the received text information using the method described in S501. The image description correction unit 813 adds or replaces information to existing image description data using the information acquired from the text information.

[0081] In the example shown in Fig. 10, text information such as "The floor is a dark color" and "The lighting is brighter" is obtained as feedback. Based on this, information such as "a dark color floor" and "bright lighting" can be added to the image description data to make it closer to the image that will actually be captured. Based on the image description data corrected in this manner, the processing from S404 onwards in Fig. 4 is executed. In other words, the image generation unit 306 regenerates the image using the image description data corrected according to the flowchart in Fig. 9.

[0082] According to the present embodiment, the image description data can be corrected by receiving user feedback on the generated images. This allows the video analysis model to be trained using images similar to those seen when the user device is in operation, without requiring the user to provide images actually taken with the user device. In other words, highly accurate video analysis can be performed from the very start of the user device's installation.

[0083] Other Embodiments In the above-described embodiments, the image generation unit 306 generates images based solely on image description data. However, images may also be generated by using existing images posted on the news site 107, for example. For example, assume that the image description generation unit 304 acquires an image 1101, such as that shown in FIG. 11 , from a web page stored as usage condition-related information. The image 1101 shows a person 1102 who has fallen. If the posture of the person who has fallen can be reflected in the training image, the accuracy of the fall detection function can be expected to improve. Therefore, the image description generation unit 304 acquires information on the joint points of the person 1102 (circles 1103 indicate joint points, and dotted lines indicate line segments connecting the joint points) using a method such as OpenPose. Next, using the acquired joint point information as a condition, the image is generated using a method such as ControlNet, such that the posture of the person in the image is similar. In this way, the image generation unit 306 generates an image that reflects the characteristics of the acquired image 1101. Here, since a fall detection function is assumed, the joint points of a person are acquired as image features, but the features acquired from the image are not particularly limited as long as they are features related to the video analysis function. For example, the image generation unit 306 may generate an image with a similar background using the results of dividing the image into units such as walls and floors. By generating an image using an existing image in this way, it is possible to generate an image with features similar to the image being used in combination.

[0084] A storage medium storing a program for implementing the functions of the learning device 111 according to each of the above-described embodiments may be supplied to a system or device, and the computer (or CPU, MPU, or GPU) of the system or device may read and execute the program code stored in the storage medium. Execution of the program code read by the computer not only realizes the functions of the above-described embodiments, but also includes an operating system (OS) running on the computer performing some or all of the actual processing based on the instructions of the program code.

[0085] Furthermore, the functions of the learning device 111 according to the above-described embodiment may be realized by the following method. Program code read from a storage medium is written to memory provided on a function expansion card inserted into a computer or a function expansion unit connected to the computer. Then, based on the instructions of the program code, a CPU or the like provided on the function expansion card or function expansion unit performs some or all of the actual processing. The storage medium stores program code corresponding to the flowchart described above.

[0086] Although the present invention has been described above with reference to the embodiments, the above embodiments are merely illustrative of specific examples of how the present invention can be implemented, and the technical scope of the present invention should not be construed as being limited by these embodiments. In other words, the present invention can be implemented in various forms without departing from its technical concept or main features.

[0087] The present invention can also be realized by supplying a program that realizes one or more of the functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., an ASIC) that realizes one or more of the functions.

[0088] The disclosure of each of the above-described embodiments includes the following configurations, methods, and programs.

[0089] (Configuration 1) An image generation device comprising: an acquisition means for acquiring related information related to a usage condition of a video analysis based on the usage condition of the video analysis; a description generation means for generating, based on the related information, description data representing an image that is expected to be the subject of the video analysis when the video analysis is used; and an image generation means for generating, based on the description data, an image for training a model to be used in the video analysis.

[0090] (Configuration 2) The image generating device described in Configuration 1, characterized in that the acquisition means acquires at least one of the following information as the usage conditions: the name of the store where the camera that captures the video to be analyzed is installed, the type of business in which the camera is used, and the location where the camera is installed; and acquires the related information based on the usage conditions.

[0091] (Configuration 3) The image generating device according to Configuration 1, wherein the acquisition means acquires at least one of the type of camera that captures the video to be analyzed, the camera's shooting resolution, the camera's shooting time, and the type of video analysis as the usage conditions, and acquires the related information based on the usage conditions.

[0092] (Configuration 4) The image generating device according to any one of configurations 1 to 3, wherein the description generating means acquires item information corresponding to a predetermined description item from the related information, and generates the description data written in a format corresponding to the function of the video analysis based on the item information.

[0093] (Configuration 5) The image generating device according to configuration 2, wherein the predetermined description items are items relating to at least one of an object in the background of the image, lighting conditions, a photographing range, and image quality.

[0094] (Configuration 6) The image generating device according to configuration 2 or 3, wherein the predetermined description items are items that represent attributes of a subject to be analyzed.

[0095] (Configuration 7) The image generating device according to any one of configurations 1 to 6, wherein the acquiring means searches a website for a web page related to the terms of use, and acquires the related information from text information included in the searched web page.

[0096] (Configuration 8) The image generating device according to Configuration 7, wherein the image generating means generates the learning images based on the description data and features extracted from existing images included in the searched web pages.

[0097] (Configuration 9) The image generating device according to any one of configurations 1 to 8, wherein the acquisition means acquires the related information from a server managed by a user who uses the video analysis.

[0098] (Configuration 10) The image generating device according to any one of configurations 1 to 9, further comprising a learning means for learning a model to be used in the video analysis using the learning images generated by the image generating means.

[0099] (Configuration 11) The image generating device according to any one of configurations 4 to 6, wherein the description generating means increases the variation of the description data by detailing the item information.

[0100] (Configuration 12) The image generating device according to any one of configurations 4 to 6 and 11, characterized in that the description generating means controls to adjust the ratio of the description data including the specific item information to the entirety of the generated description data.

[0101] (Configuration 13) The image generating device according to any one of configurations 1 to 12, wherein the description generating means controls to adjust the description data generated from the related information according to the importance of the related information.

[0102] (Configuration 14) The image generating device according to any one of configurations 1 to 13, further comprising: a presentation means for presenting the learning images generated by the image generating means to a user using the video analysis; a reception means for receiving feedback on the presented learning images; and a correction means for correcting the description data based on the feedback.

[0103] (Configuration 15) The image generating device according to configuration 14, wherein the presentation means presents a screen on which the learning image and an input field for a comment on the learning image are displayed.

[0104] (Configuration 16) The image generating device according to configuration 10, further comprising a transmitting means for transmitting the model learned by the learning means to a device that performs the video analysis under the usage conditions.

[0105] (Configuration 17) The image generating device according to any one of configurations 1 to 16, wherein the model is a neural network.

[0106] (Configuration 18) A video analysis system including a video analysis device that performs video analysis of video captured by a camera, and a learning device that learns a model to be used for the video analysis, wherein the learning device has: an acquisition means that acquires related information related to the usage conditions of the video analysis device based on the usage conditions; a description generation means that generates description data representing images that are expected to be the subject of video analysis by the video analysis device based on the related information; an image generation means that generates images based on the description data; and a learning means that learns a model to be used for the video analysis using the images generated by the image generation means; and the video analysis device has: an analysis means that performs the video analysis using the model learned by the learning means.

[0107] (Method) An image generation method comprising: an acquisition step of acquiring related information related to the usage conditions of the video analysis based on the usage conditions of the video analysis; a description generation step of generating, based on the related information, description data representing images that are expected to be the subject of the video analysis when the video analysis is used; and an image generation step of generating, based on the description data, images for training a model to be used in the video analysis.

[0108] (Program) A program that causes a computer of an image generation device to function as: an acquisition means that acquires related information related to the usage conditions of the video analysis based on the usage conditions of the video analysis; a description generation means that generates, based on the related information, description data representing images that are expected to be the subject of the video analysis when the video analysis is used; and an image generation means that generates, based on the description data, images for training a model to be used in the video analysis.

[0109] The present invention is not limited to the above-described embodiments, and various modifications and variations can be made without departing from the spirit and scope of the present invention. Therefore, the following claims are appended to apprise the public of the scope of the present invention.

[0110] This application claims priority based on Japanese Patent Application No. 2023-161598, filed September 25, 2023, the entire contents of which are incorporated herein by reference.

[0111] 102 In-house mail server 103 Sales data management server 104 Video analysis device 106 User site 107 News site 108 Information site 111 Learning device 112 Internet

Claims

1. An image generation device comprising: an acquisition means for acquiring related information related to a usage condition of the video analysis based on the usage condition of the video analysis; a description generation means for generating, based on the related information, description data representing an image that is expected to be the subject of the video analysis when the video analysis is used; and an image generation means for generating an image to be used in the video analysis based on the description data.

2. The image generating device described in claim 1, characterized in that the acquisition means acquires at least one of the following information as the conditions of use: the name of the store in which the camera that captures the image to be analyzed is installed, the type of business in which the camera is used, and information regarding the location in which the camera is installed, and acquires the related information based on the conditions of use.

3. The image generating device described in claim 1, characterized in that the acquisition means acquires at least one of the type of camera that captures the image to be analyzed, the shooting resolution of the camera, the shooting time of the camera, and the type of image analysis as the usage conditions, and acquires the related information based on the usage conditions.

4. The image generating device described in claim 1, characterized in that the description generation means obtains item information, which is information corresponding to a specified description item, from the related information, and generates the description data written in a format corresponding to the function of the video analysis based on the item information.

5. The image generating device according to claim 4, wherein the predetermined description items are items relating to at least one of the following: objects in the background of the image, lighting conditions, shooting range, and image quality.

6. An image generating apparatus according to claim 4, wherein the predetermined description items are items expressing attributes of a subject to be analyzed.

7. The image generating device according to claim 1, wherein said acquisition means searches a website for a web page related to said terms of use, and acquires said related information from text information contained in the searched web page.

8. The image generating device according to claim 7, characterized in that the image generating means generates the learning images based on the descriptive data and features extracted from existing images contained in the searched Web pages.

9. The image generating device according to claim 1, characterized in that the acquisition means acquires the related information from a server managed by a user who uses the video analysis.

10. The image generating device according to claim 1, further comprising a learning means for learning a model used in the video analysis using the images generated by the image generating means.

11. The image generating device according to claim 4, wherein said description generating means increases the variation of said description data by detailing said item information.

12. The image generating device according to claim 4, characterized in that said description generating means controls to adjust the ratio of said description data including said specific item information to the entirety of said generated description data.

13. The image generating apparatus according to claim 1, wherein said description generating means adjusts said description data generated from said related information in accordance with the importance of said related information.

14. The image generating device according to claim 1, further comprising: a presentation means for presenting the image generated by the image generating means to a user using the video analysis; a reception means for receiving feedback on the presented image; and a correction means for correcting the description data based on the feedback.

15. The image generating device according to claim 14, wherein said presentation means presents a screen on which the image and an input field for a comment on the image are displayed.

16. An image generating device according to claim 10, further comprising a transmitting means for transmitting the model learned by the learning means to a device that performs the video analysis in the usage environment.

17. The image generating apparatus of claim 1, wherein the model includes a neural network.

18. A video analysis system including a video analysis device that performs video analysis of video captured by a camera, and a learning device that learns a model to be used for the video analysis, wherein the learning device has: an acquisition means for acquiring related information related to usage conditions of the video analysis device based on the usage conditions; a description generation means for generating description data representing images that are expected to be the subject of video analysis by the video analysis device based on the related information; an image generation means for generating images based on the description data; and a learning means for learning the model using the images generated by the image generation means; and the video analysis device has: an analysis means for performing the video analysis using the model learned by the learning means.

19. An image generation method comprising: an acquisition step of acquiring related information related to the usage conditions of the video analysis based on the usage conditions of the video analysis; a description generation step of generating, based on the related information, description data representing an image that is expected to be the subject of the video analysis when the video analysis is used; and an image generation step of generating, based on the description data, an image to be used in the video analysis.

20. A program that causes a computer of an image generation device to function as: an acquisition means that acquires, based on the usage conditions of the video analysis, relevant information related to the conditions; a description generation means that generates, based on the relevant information, description data representing images that are expected to be the subject of the video analysis when the video analysis is used; and an image generation means that generates images to be used in the video analysis based on the description data.

Images