Image generation device, video analysis system, image generation method, and program

Abstract

To enable tuning of a camera to be started faster than additional learning on the basis of test data obtained after installing the camera in a specific location.SOLUTION: A learning device 111 acquires usage condition-related information associated with the usage conditions from servers 102, 103 of a user environment 101 or Websites 106, 107, 108 in a Web environment 105 on the basis of usage conditions of video analysis. From the usage condition-related information, the learning device 111 extracts item information corresponding to each image description item and generates image description data representing an image that is expected to be subject for image analysis when using video analysis using the extracted item information. Further, images for learning a model used for video analysis are generated on the basis of the image description data.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to machine learning. [Background technology]

[0002] In recent years, the application of technology to analyze images from surveillance cameras has progressed. For example, a technology to estimate the posture of a person's entire body has been proposed, and applications are progressing in the fields of customer protection in stores and urban surveillance. Machine learning, particularly learning models such as neural networks, are widely used in image analysis technology. Patent Document 1 describes a method of fine-tuning a trained neural network model using test data (images and labels) acquired by a user-side device. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2018 / 142766 Summary of the Invention [Problem to be solved by the invention]

[0004] In a neural network for video analysis, if the tendency of the input video differs between when it is learned (during development) and when it is used by a user (during operation), sufficient analysis accuracy may not be obtained. Examples of the tendency of the input video include the shooting environment such as lighting conditions, and statistical trends of the attributes of the analysis target such as people (e.g., most people in their 50s). In the method using test data as described in Patent Document 1, additional learning must be performed using the video obtained by shooting at the user's installation location for a certain period of time. In other words, there is a problem that sufficient analysis accuracy cannot be obtained during the test data creation period.

[0005] In view of the above problems, an object of the present invention is to enable tuning of a camera to begin sooner than with a method in which additional learning is performed based on test data obtained after the camera is installed in a specific location. [Means for solving the problem]

[0006] The image generation device of the present invention is characterized in that it has an acquisition means for acquiring related information related to the usage conditions of the video analysis based on the usage conditions of the video analysis, a description generation means for generating description data representing an image that is expected to be the subject of the video analysis when the video analysis is used based on the related information, and an image generation means for generating an analysis image to be used in the video analysis based on the description data. Effect of the Invention

[0007] According to the present invention, it is possible to start tuning of the camera earlier than with a method in which additional learning is performed based on test data obtained after the camera is installed at a specific location. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of the overall configuration of a video analysis system. [Diagram 2] FIG. 2 illustrates an example of a hardware configuration of a learning device. [Diagram 3] FIG. 2 is a diagram illustrating an example of a functional configuration of a learning device. [Figure 4] 13 is a flowchart showing a process executed by the learning device. [Diagram 5] 13 is a flowchart showing details of an image description data generation process. [Figure 6] FIG. 11 is a diagram showing an example of order information. [Figure 7] FIG. 13 is a diagram showing an example of a table in which sub-items are defined. [Figure 8] FIG. 2 is a diagram illustrating an example of a functional configuration of a learning device. [Figure 9]13 is a flowchart showing details of an image description data correction process. [Figure 10] FIG. 13 is a diagram showing an example of a feedback screen. [Figure 11] FIG. 13 is a diagram showing an example of an image used for learning. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, each embodiment will be described with reference to the accompanying drawings.

[0010] <Embodiment 1> FIG. 1 shows an example of a video analysis system according to this embodiment. The video may be a video or a still image. The following mainly describes an example in which the video is a video. Video analysis is a process of extracting a subject to be analyzed from information captured in the video, and identifying the movement and position of the subject, thereby outputting an analysis result according to the video analysis function. The following mainly describes an example in which the subject is a person. In this embodiment, video analysis is performed by inputting the video into a video analysis model.

[0011] 1, the video analysis system includes various devices provided in a user environment 101 of a user who uses a video analysis device 104, and a learning device 111 provided in a vendor environment 110 of a developer who develops a video analysis model. The various devices provided in the user environment 101 and the learning device 111 are connected to the Internet 112. Various websites provided in a Web environment 105 and a search server 109 are connected to the Internet 112. The learning device 111 transmits and receives data to and from the various devices connected to the Internet 112.

[0012] The video analysis device 104 is a device that integrates a camera (for example, a surveillance camera) that captures video and a device that inputs the captured video into a video analysis model to perform video analysis. The video analysis device 104 also has functions according to the purpose of the video analysis. The learning device 111 optimizes the video analysis model according to the functions. In this embodiment, the video analysis device 104 has a fall detection function, and when the video analysis device 104 is in operation, the fall detection function is used to detect a person's fall from the captured video (input video). Note that the video analysis device 104 may have a purchase analysis function, an intrusion detection function, a crowd detection function, a suspicious behavior detection function, and the like, instead of or in addition to the fall detection function. In this embodiment, the video analysis device 104 is used in a store, but it may also be used outdoors, such as in buildings other than stores or in town.

[0013] In this embodiment, the learning device 111 acquires information related to the use conditions (use condition related information) from the use conditions of the video analysis device 104, and generates an image (analysis image) representing an input video expected when the video analysis device 104 is in operation based on the acquired use condition related information. Then, the learning device 111 uses the image (analysis image) as a learning image for a video analysis model. In other words, before the operation of the video analysis device 104 begins, a learning image close to that during operation can be prepared. The learning device 111 is an example of an image generation device.

[0014] The user environment 101 includes an in-house mail server 102, a sales data management server 103, a video analysis device 104, and a PC 115, all of which are used by the user. The in-house mail server 102 and the sales data management server 103 store various data related to the user's business and operations. The PC (personal computer) 115 is a terminal device used by the user. The PC 115 displays a UI screen received from the learning device 111 on a display device (not shown), and transmits information input to the UI screen via an input device (not shown) to the learning device 111. The Web environment 105 includes user sites 106 established by users themselves, news sites 107 established by news organizations, and information sites 108 that compile various information. The search server 109 searches various websites provided in the Web environment 105 using the query (keywords) received from the learning device 111, and transmits the search results (such as a URL) to the learning device 111.

[0015] 2 is a block diagram showing the hardware configuration of learning device 111. Learning device 111 has a CPU 201, ROM 202, RAM 203, secondary storage device 204, input device 205, display device 206, and network I / F 207. These components are connected via bus 208 to input and output data to and from each other.

[0016] The CPU (Central Processing Unit) 201 is responsible for the overall control of the learning device 111. Note that a GPU (Graphics Processing Unit) may be used instead of or together with the CPU. The ROM 202 is a non-volatile memory and stores various programs and various data. The RAM 203 is a volatile memory and stores frame image data and temporary data generated in the flowcharts described below. The secondary storage device 204 is a rewritable storage device such as a hard disk drive or flash memory and stores image information, programs, various setting information, and the like.

[0017] The input device 205 is a keyboard, mouse, or the like, and inputs operations from the developer. The display device 206 is a liquid crystal display, or the like, and displays processing results, etc. to the developer. The network I / F 207 is a modem, LAN, or the like that connects to a network such as the Internet 112 or an intranet. The CPU 201 reads out programs stored in the ROM 202, secondary storage device 204, or the like into the RAM 203 and executes them, thereby realizing processing corresponding to each step of a flowchart described later.

[0018] Fig. 3 is a block diagram showing the functional configuration of the learning device 111 according to this embodiment. The CPU 201 reads out a program stored in the ROM 202, the secondary storage device 204, etc., into the RAM 203 and executes it, thereby realizing each function shown in Fig. 3.

[0019] The usage condition input unit 301 inputs the usage conditions of the video analysis device 104 used by the user. The related information acquisition unit 302 collects information related to the use conditions inputted in the use condition input unit 301 (use condition related information) from the user environment 101 and the Web environment 105 via the network I / F 207 . The related information storage unit 303 stores the use condition related information collected by the related information acquisition unit 302 in the secondary storage device 204, the RAM 203, or the like. The image description generating unit 304 generates image description data based on the use condition related information stored in the related information storage unit 303 . The image description storage unit 305 stores the image description data generated by the image description generation unit 304 in the secondary storage device 204, the RAM 203, or the like.

[0020] The image generation unit 306 generates an image and a correct label to be assigned to the image based on the image description data stored in the image description storage unit 305 . The generated image storage unit 307 stores the generated image and the correct label generated by the image generation unit 306 in the secondary storage device 204, the RAM 203, or the like. The trained model storage unit 308 stores a trained video analysis model corresponding to a function (here, a fall detection function) of the video analysis device 104 in the secondary storage device 204 or the like. As the video analysis model, a known machine learning method such as a neural network is used. The additional learning unit 309 performs additional learning of the video analysis model stored in the trained model storage unit 308 using the generated images and correct labels stored in the generated image storage unit 307. The model providing unit 310 transmits the video analysis model obtained by additional learning to the video analysis device 104 via the network I / F 207.

[0021] 4 is a flowchart showing the details of the processing executed by the learning device 111 according to this embodiment. In the following description, each process (step) is represented by adding an S to the beginning, and the notation of the process (step) is omitted.

[0022] First, in S401, the use condition input unit 301 acquires order information for the video analysis device 104. Then, the CPU 201 extracts the use conditions of the video analysis device 104 used by the user from the acquired order information, and acquires the extracted use conditions.

[0023] FIG. 6 shows an example of order information when the video analysis device 104 is used in a store. The order information is information generated when the developer receives an order for the video analysis device 104 from a user, and is stored in, for example, a data server installed in the vendor environment 110. The use condition input unit 301 acquires the order information from the data server via the network I / F 207. The order information in FIG. 6 includes contents corresponding to various items such as an order recipient 601, a function 602, a business type 603, a store used 604, a camera installation location 605, a camera used 606, a camera setting such as a set resolution 607, an operation time 608, a price 609, and a delivery date 610. The order recipient 601 indicates the name of the store used. The function 602 indicates a function of the video analysis according to the purpose of use of the video analysis device 104. The business type 603 indicates the form of the store used. The store used 604 indicates the name of the branch of the store used. The camera used 606 indicates the model information of the camera that shoots the video.

[0024] In the following description, it is assumed that, from among the items of the order information, contents corresponding to predetermined items related to the use conditions (items 601 to 610 in the example of FIG. 6) are extracted as the use conditions. Note that, among the items 601 to 610, only contents corresponding to some of the items may be extracted. Furthermore, without being limited to the method of extracting the use conditions from the order information, the use condition input unit 301 may transmit a UI screen (not shown) for inputting the use conditions to the PC 115, and obtain information input by the user via the UI screen as the use conditions.

[0025] In S402, the related information acquisition unit 302 collects use condition related information based on the use conditions acquired in S401. The use condition related information includes the conditions of an image that is expected to be captured when an image is captured under the use conditions acquired in S401, and information for identifying the conditions. The image description storage unit 305 stores the collected use condition related information in the secondary storage device 204, the RAM 203, etc. as needed.

[0026] The information related to the usage conditions relates to the background of the image. For example, it relates to background objects, lighting conditions, shooting range, and image quality. The background objects are the types of objects expected to exist in the background, for example, product shelves in a store, and roads in a city. The lighting conditions are information specifying the sunlight conditions (weather and time of use) if outdoors, and the type of light source and window size if indoors. The influence of external light can be specified from the window size. The shooting range is, for example, lens information of the camera used (whether wide-angle or telephoto, focal length, etc.), and the installation position and angle of the camera. The image quality is the image quality of the entire image, for example, the performance and set resolution of the camera used. Note that there is no particular limitation as long as it is information related to the background of the image. The usage condition-related information relates to the attributes of people in the image, for example, the characteristics of people expected to appear in the video (such as their age, sex, race, clothing, and posture).

[0027] The related information acquisition unit 802 searches the Web environment 105 or the user environment 101 for information related to the background of the image and the attributes of the people in the image as described above, and acquires information on the Web page obtained as a result of the search as usage condition related information.

[0028] When obtaining the use condition related information from the Web environment 105 , the related information obtaining unit 802 obtains the use condition related information from the Web environment 105 by using the search server 109 . The related information acquisition unit 802 performs a search using, for example, the function 602 "fall detection" as a query. As a result of the search, the related information acquisition unit 802 acquires, for example, article information on fall fraud from the news site 107. Assume that this article information includes text information such as "Fall fraud has occurred. A white man in his 50s wearing a dress shirt fell over himself in a beverage counter and sued the store." The related information acquisition unit 802 also performs a search using, for example, "A super" of the order recipient 601 as a query. As a result of the search, the related information acquisition unit 802 acquires, for example, basic store information such as sales floor configuration and business hours from the user site 106 of A super. The related information acquisition unit 802 also performs a search using, for example, the "Las Vegas store" of the store of use 604 as a query. The related information acquisition unit 802 acquires, for example, the racial and age demographic composition ratio of Las Vegas from the information site 108 as a search result. Furthermore, the related information acquisition unit 802 performs a search using, for example, the model name "M30" of the camera 606 used as a query. As a result of the search, the related information acquisition unit 802 acquires, for example, a review article of the M30 from the information site 108. Assume that this review article contains text information about image quality such as "High image quality during the day, but high noise at night."

[0029] When obtaining use condition related information from the user environment 101, the related information obtaining unit 802 obtains the use condition related information from the user environment 101 using a search server (not shown) that can access the in-house mail server 102 and the sales data management server 103.

[0030] The related information acquisition unit 802 acquires testimony information from other stores from the in-house mail server 102, for example, "A man in his 50s sued the New York store, claiming that he fell in the vegetable section and was injured." Furthermore, the related information acquisition unit 802 acquires data on person attributes, such as "50% of the purchasing demographic are in their 40s, 40% in their 50s" from the sales data management server 103.

[0031] In S403, the image description generating unit 304 generates image description data based on the usage condition related information acquired in S402. The image description data is data for generating an image that is expected to be captured when the video analysis device 104 is used, and is described in text. In this embodiment, the image description data is generated according to a format defined for each video analysis function.

[0032] Below are examples of image description data templates for each video analysis function. 1.Fall detection function Generate an image with {image quality} of a person of {race}, {age}, {gender}, wearing {clothing}, falling at {location} at {time}. 2. Gradient analysis function Generate an image with {image quality} of a person of {race}, {age}, {gender}, wearing {clothes}, stretching their hand at {location} at {time}. 3. Intrusion detection function Generate an image with {image quality} of a person of {race}, {age}, {gender}, wearing {clothing}, entering {area} of {location} at {time} The race, age, etc. in the above {} represent image description items. The image description generation unit 304 extracts information corresponding to each image description item from the usage condition related information acquired in S402, and generates image description data in multiple variations using the extracted information and a template according to the video analysis function. Hereinafter, information corresponding to the image description items may be referred to as item information. Note that the image description items shown in 1 to 3 above in the image description data are examples, and are not particularly limited as long as they relate to the background of the image and the attributes of the subject (here, a person) to be analyzed.

[0033] Note that the above image description data examples 1 to 3 describe positive cases where a detection target exists, but depending on the function, image description data for negative cases where a detection target is not present may also be necessary. For example, in the case of the fall detection function in 1, image description data for negative cases can be generated by changing the description "falling" to other content such as "walking" or "standing."

[0034] The process of generating image description data executed in S403 will be described in detail below with reference to the flowchart of FIG. In S501, the image description generating unit 304 obtains information corresponding to each image description item from the use condition related information obtained in S402. Specifically, the image description generating unit 304 removes unnecessary information such as HTML tags from a Web page stored as use condition related information in the secondary storage device 204, the RAM 203, etc., and extracts text information. Next, if the extracted text information is in Japanese, it is broken down into parts of speech using morphological analysis, and information corresponding to each image description item is extracted by matching with a dictionary that further classifies nouns into types such as race, age, and sex. Also, information on each image description item may be extracted using a large-scale language model such as Generative Pre-trained Transformer (GPT).

[0035] For example, from the aforementioned article information "Fall fraud occurred. A white man in his 50s wearing a dress shirt fell over himself in the beverage section and sued the store," the following items of information are acquired about the person's attributes: age "50," clothing "dress shirt," race "white," and gender "male." In addition, the location where the image was taken, "beverage section," is acquired as item information about the background of the image. In addition, from the review article "High quality during the day, high noise at night," the time "daytime," image quality "high quality," and time "nighttime," image quality "high noise" are obtained as item information related to the background of the image. Based on this, the image description generating unit 304 generates the following image description data: “Generating a high-quality image of a white man in his 50s wearing a dress shirt falling in a beverage counter in daytime” “Generating a high-noise image of a white man in his 50s wearing a dress shirt falling at a beverage counter at night”

[0036] Next, in S502, the image description generating unit 304 performs detailing of information corresponding to each image description item. Since the information corresponding to the image description items is extracted from the usage condition related information, it shows contents close to the images actually shot by the user. Therefore, it is desirable to secure as many variations of images as possible generated from the information corresponding to the image description items. Therefore, the image description generating unit 304 performs detailing of the information (item information) corresponding to each image description item, thereby increasing the variation of the image description data.

[0037] Specifically, the image description generating unit 304 adds information corresponding to sub-items to the item information. Sub-items are lower-level items for detailing the item information. For example, when "beverage counter" is extracted as "photography location", the image description generating unit 304 acquires "area" in the counter as a sub-item, and adds "in front of the shelf", "aisle", etc. corresponding to the "area". Also, for example, when "shirt" is extracted as "clothing", "color" and "pattern" are acquired as sub-items of the shirt, and "white", "black", "red", etc. corresponding to the "color", and "solid", "stripes", "check", etc. corresponding to the "pattern". FIG. 7 shows an example of a table that defines sub-items. The image description generating unit 304 acquires sub-items, which are lower-level items of the item information, and information corresponding to each sub-item by referring to a table such as that shown in FIG. 7.

[0038] For example, the image description generating unit 304 generates the following variations of image description data by refining the shooting location "drink counter" and the clothing "dress shirt" for the above image description data. “We generated a high-resolution image of a white male in his 50s wearing a plain white shirt, slumped in front of a beverage shelf in the daytime.” “We generated a high-resolution image of a white male in his 50s wearing a plain black shirt, slumped in front of a beverage shelf in the daytime.” “We generated a high-resolution image of a white male in his 50s wearing a plain red shirt, slumped in front of a beverage shelf in the daytime.” “Generating a high-resolution image of a white male in his 50s wearing a white striped shirt slumped in front of a beverage shelf in daytime.” (The rest is omitted.) As described above, by combining sub-items with information corresponding to each image description item and acquiring information corresponding to the sub-items, it is possible to increase the variety of image description data.

[0039] Here, the use condition related information includes information that is highly likely to be actually photographed (highly credible), such as the above-mentioned testimony information from other stores. The image description generating unit 304 may adjust the information acquired from such use condition related information with high importance so that more variations are generated. The importance here is determined, for example, according to the source from which the use condition related information is acquired. The image description generating unit 304 may only target image description items acquired from testimony information for detailing, or may increase the number of sub-items for image description items acquired from testimony information compared to when the image description items are acquired from sources other than testimony information. In this way, the image description generating unit 304 adjusts the number of image description data generated based on the use condition related information according to the importance of the use condition related information.

[0040] Also, for example, information on percentages such as "40s 50% and 50s 40%" may be extracted from the use condition related information acquired from the sales data management server 103 or the information site 108. In such a case, it is considered that the people expected to appear in the video will have a similar percentage. Therefore, the image description generating unit 304 may adjust the percentages of image description data for ages "40s" and "50s" in the total image description data to be generated so that they are 50% and 40%, respectively.

[0041] Next, in S503, the image description generating unit 304 performs adjustment according to the image description item. The use condition related information represents typical contents of the video that is actually shot, and therefore has a bias such that certain image description items contain a lot of specific information. However, some image description items may cause inconvenience such as ethical problems by making it easier to detect people with specific attributes such as race or sex. To address this, the image description generating unit 304 may perform adjustment so that the ratio of image description data in which a specific image description item is specific information to the entire generated image description data does not exceed a threshold value. Specifically, if the ratio is equal to or exceeds the threshold value, the generated image description data is randomly thinned out until it falls below the threshold value. The image description generating unit 304 stores the plurality of image description data generated as described above in the secondary storage device 204, the RAM 203, etc. In this manner, a series of processes for generating image description data is completed.

[0042] Returning to the explanation of FIG. 4, in S404, the image generation unit 306 generates a learning image (analysis image) based on the image description data generated in S403. Specifically, a model called Contrastive Language-Image Pre-training (CLIP) that is pre-trained so that text and images can be expressed by a common vector is used. The image generation unit 306 encodes the image description data using CLIP as a text encoder, and uses a diffusion model or the like as conditioning when generating an image. This makes it possible to obtain an image that conforms to the image description data. Since there may be multiple images that are similar to the encoded image description data, multiple images can be obtained from one image description data.

[0043] Next, in S405, the image generation unit 306 generates a correct answer label for the learning image generated in S404 by using the image description data. This correct answer label is defined for each template of the image description data. For the image description data "falling" in 1 above, "falling" is defined as the correct answer label, and for the negative examples "walking" and "standing", "not falling" is defined as the correct answer label. The generated image storage unit 307 assigns correct answer labels to the multiple generated images generated in S404 and stores them in the secondary storage device 204, RAM 203, etc.

[0044] Next, in S406, the additional learning unit 309 reads out a trained video analysis model from the secondary storage device 204 or the like. The secondary storage device 204 stores video analysis models that have been trained in advance using existing images held by the developer for each video analysis function. The additional learning unit 309 reads out a video analysis model that corresponds to the function to be used (in this embodiment, fall detection).

[0045] In S407, the additional learning unit 309 performs additional learning of the video analysis model read out in S406 based on the generated image and the correct label stored in S405. Specifically, the CPU 201 may fix the weight of the front part (backbone) that extracts basic features of the model with the weight of the trained model as the initial value, and update only the weight of the rear part. Of course, the method is not limited to the above, and any learning method that adapts the model to the generated image, such as updating the weight of the entire model, may be used.

[0046] In S408, the model providing unit 310 judges whether or not the introduction of the video analysis device 104 has started on the user side. Specifically, the model providing unit 310 checks based on a notification from the user by e-mail or the like. The process waits in S408 until the model providing unit 310 judges that the introduction of the video analysis device 104 has started, and when it is judged that the introduction of the video analysis device 104 has started, the process proceeds to S409. In S409, the model providing unit 310 transmits the updated model that has been additionally trained in S407 to the video analysis device 104. For example, the video analysis device 104 may download the updated model from the learning device 111 via the Internet 112. The video analysis device 104 inputs the video captured by the camera into the video analysis model provided by the learning device 111 and performs video analysis. After that, the series of processes in the flowchart shown in FIG. 4 ends.

[0047] According to the present embodiment described above, image description data representing images that are expected to be actually captured can be generated from information related to the usage conditions of the user's video analysis device, and learning images can be generated based on the image description data. This makes it possible to learn a video analysis model using images similar to those seen when the user's device is in operation, without the need to receive images actually captured by the user's device. In other words, highly accurate video analysis can be performed from the start of operation of the user's device.

[0048] <Embodiment 2> In the first embodiment, a method was described in which learning images generated based on image description data are used as is for additional learning. In the second embodiment, a method is described in which a user is asked to check learning images generated based on image description data, feedback from the user is reflected, and then the images are used for additional learning. This makes it possible to generate learning images that are more suited to the usage environment. Note that a description of the points common to the first embodiment will be omitted, and the differences from the first embodiment will be mainly described.

[0049] Fig. 8 is a block diagram showing the functional configuration of the learning device 111 according to this embodiment. A use condition input unit 801, a related information acquisition unit 802, a related information storage unit 803, and an image description generation unit 804 in Fig. 8 correspond to the use condition input unit 301, the related information acquisition unit 302, the related information storage unit 303, and the image description generation unit 304 in Fig. 3. Also, an image description storage unit 805, an image generation unit 806, and a generated image storage unit 807 in Fig. 8 correspond to the image description storage unit 305, the image generation unit 306, and the generated image storage unit 307 in Fig. 3. Also, a trained model storage unit 808, an additional learning unit 809, and a model providing unit 810 in Fig. 8 correspond to the trained model storage unit 308, the additional learning unit 309, and the model providing unit 310 in Fig. 3.

[0050] A generated image providing unit 811 transmits the image generated by the image generating unit 806 to the user's PC 115 via the network I / F 207 . The feedback receiving unit 812 receives text information that is feedback input by the user on the PC 115 . The image description correcting unit 813 corrects the image description data stored in the image description storage unit 805 .

[0051] The basic process flow executed by the learning device 111 according to this embodiment is similar to the flowchart shown in Fig. 4. Hereinafter, the process flow of performing correction based on feedback from a user, which is a characteristic process of this embodiment, will be described with reference to Fig. 9.

[0052] After an image is generated in S404 in Fig. 4, in S901, the generated image providing unit 811 selects a generated image to be provided to the user. Since it is not realistic to have the user check all generated images, the generated images are selected and provided. Note that the method of selecting images may be any method, such as a method of randomly selecting a predetermined number of images, or a method of selecting only images generated based on image description data in which a predetermined image description item is specific information.

[0053] Next, in S902, generated image providing unit 811 transmits the generated image selected in S901 to PC 115 via Internet 112. Note that generated image providing unit 811 does not provide the generated image as is, but transmits data in HTML format for displaying a confirmation UI screen having a screen configuration as shown in Fig. 10 on PC 115 in order to receive feedback from the user. In this way, generated image providing unit 811 presents to the user a confirmation UI screen on which the generated image selected in S901 is displayed.

[0054] 10 is an example of a screen configuration of a confirmation UI screen presented to a user. The confirmation UI screen is displayed on a display device of PC 115. Images 1002 and 1003 (two images in this example) selected in S901 are displayed on confirmation UI screen 1001. Images 1002 and 1003 are images generated from image description data. Input fields 1004 and 1005 are arranged to the right of images 1002 and 1003, where a user can input a comment in text. When a user inputs feedback in input fields 1004 and 1005 via an input device of PC 115 and then presses send button 1006, PC 115 transmits the contents (text information) of input fields 1004 and 1005 to learning device 111 via Internet 112.

[0055] Next, in S903, the feedback receiving unit 812 determines whether or not feedback (text information) has been received from the PC 115. The feedback receiving unit 812 waits in S903 until it receives feedback, and if it determines that the feedback receiving unit 812 has received feedback, the process proceeds to S904.

[0056] In S904, the image description correction unit 813 corrects the image description data stored in the secondary storage device 204, the RAM 203, etc., based on the text information received in S903. The image description correction unit 813 acquires information corresponding to each image description item from the received text information using the method described in S501. The image description correction unit 813 adds or replaces information to the existing image description data using the information acquired from the text information.

[0057] In the example shown in Fig. 10, text information such as "The floor is a dark color" and "The lighting is brighter" is obtained as feedback. Based on this, information such as "dark color floor" and "bright lighting" can be added to the image description data to make it closer to the image that is actually captured. Based on the image description data corrected in the above manner, the processes from S404 onwards in Fig. 4 are executed. That is, the image generation unit 306 regenerates the image using the image description data corrected according to the flowchart in Fig. 9.

[0058] According to the present embodiment described above, it is possible to receive feedback from the user on the generated images and correct the image description data. This makes it possible to train a video analysis model using images similar to those seen when the user device is in operation, without the need to receive images actually taken with the user device. In other words, it is possible to perform highly accurate video analysis from the start of the introduction of the user device.

[0059] <Other embodiments> In each of the above-mentioned embodiments, the image generating unit 306 generates an image based only on the image description data, but an image may be generated by using an existing image posted on the news site 107 or the like. For example, it is assumed that the image description generating unit 304 acquires an image 1101 as shown in FIG. 11 from a Web page stored as the use condition related information. A person 1102 who has fallen is shown in the image 1101. If the posture of the person who has fallen can be reflected in the learning image, the accuracy of the fall detection function can be expected to improve. Therefore, the image description generating unit 304 acquires information on the joint points of the person 1102 (circles 1103 indicate joint points, and dotted lines indicate lines connecting the joint points) using a method such as OpenPose. Next, the image is generated by using a method such as ControlNet with the acquired information on the joint points as a condition so that the postures of the people in the image are similar. In this way, the image generating unit 306 generates an image that reflects the characteristics of the acquired image 1101. In this example, since a fall detection function is assumed, the joint points of a person are acquired as image features, but the features acquired from an image are not particularly limited as long as they are features related to the video analysis function. For example, the image generating unit 306 may generate an image with a similar background by using the results of dividing an image into units such as walls and floors. In this way, by generating an image using an existing image in combination, an image with similar features to the image being used in combination can be generated.

[0060] A storage medium storing a program for implementing the functions of the learning device 111 according to each of the above-described embodiments may be supplied to the system or device, and the computer (or CPU, MPU, GPU) of the system or device may read and execute the program code stored in the storage medium. In addition, the computer may execute the program code it reads to implement not only the functions of the above-described embodiments, but also the operating system (OS) running on the computer may execute part or all of the actual processing based on the instructions of the program code.

[0061] Furthermore, the following method may be used to realize the functions of the learning device 111 according to the embodiment described above. Program code read from a storage medium is written to a memory provided in a function expansion card inserted into a computer or a function expansion unit connected to a computer. Then, based on instructions from the program code, a CPU or the like provided in the function expansion card or function expansion unit performs some or all of the actual processing. The recording medium stores program code corresponding to the flowchart described above.

[0062] Although the present invention has been described above with reference to the embodiments, the above embodiments are merely illustrative of the specific examples of the present invention, and the technical scope of the present invention should not be interpreted as being limited by these embodiments. In other words, the present invention can be embodied in various forms without departing from its technical concept or main features.

[0063] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0064] The disclosure of each of the above-described embodiments includes the following configurations, methods, and programs. (Configuration 1) An acquisition means for acquiring related information related to a use condition based on the use condition of the video analysis; a description generating means for generating description data representing an image that is expected to be a target of the video analysis when the video analysis is performed based on the related information; an image generating means for generating an analysis image to be used in the video analysis based on the description data; An image generating device comprising: (Configuration 2) The image generating device described in configuration 1, characterized in that the acquisition means acquires at least one of the information regarding the name of the store in which the camera that captures the image to be analyzed is installed, the type of business in which the camera is used, and the location in which the camera is installed as the usage conditions, and acquires the related information based on the usage conditions. (Configuration 3) The image generating device described in configuration 1, characterized in that the acquisition means acquires at least one of the type of camera that captures the image to be analyzed, the imaging resolution of the camera, the imaging time of the camera, and the type of image analysis as the usage conditions, and acquires the related information based on the usage conditions. (Configuration 4) The image generating device of any one of configurations 1 to 3, characterized in that the description generation means acquires item information, which is information corresponding to a specified description item, from the related information, and generates the description data written in a format corresponding to the function of the video analysis based on the item information. (Configuration 5) 5. The image generating device according to configuration 4, wherein the predetermined description items are items related to at least one of an object in the background of the image, a lighting condition, a shooting range, and an image quality. (Configuration 6) 6. The image generating device according to configuration 4 or 5, wherein the predetermined description items are items that represent attributes of a subject to be analyzed. (Configuration 7) The image generating device described in any one of configurations 1 to 6, characterized in that the acquisition means searches a website for web pages related to the terms of use and acquires the related information from text information contained in the searched web pages. (Configuration 8) The image generating device according to configuration 7, wherein the image generating means generates the analysis image based on the description data and features extracted from existing images included in the searched Web pages. (Configuration 9) 9. The image generating device according to any one of configurations 1 to 8, wherein the acquisition means acquires the related information from a server managed by a user who uses the video analysis. (Configuration 10) The image generating device according to any one of configurations 1 to 9, further comprising a learning means for learning a model used in the video analysis using the analysis image generated by the image generating means. (Configuration 11) 7. The image generating device according to any one of configurations 4 to 6, wherein the description generating means increases the variation of the description data by detailing the item information. (Configuration 12) The image generating device according to any one of configurations 4 to 6 and 11, characterized in that the description generation means controls to adjust the ratio of the description data including specific item information to the entirety of the generated description data. (Configuration 13) 13. The image generating device according to any one of configurations 1 to 12, wherein the description generating means controls to adjust the description data generated from the related information according to the importance of the related information. (Configuration 14) a presentation means for presenting the analysis image generated by the image generation means to a user who uses the video analysis; a receiving means for receiving feedback on the presented analysis image; a correction means for correcting the description data based on the feedback; 14. The image generating device according to any one of configurations 1 to 13, further comprising: (Configuration 15) 15. The image generating device according to claim 14, wherein the presentation means presents a screen on which the image for analysis and an input field for a comment on the image for analysis are displayed. (Configuration 16) 11. The image generating device according to configuration 10, further comprising a transmitting means for transmitting the model learned by the learning means to a device that performs the video analysis in the usage environment. (Configuration 17) 17. The image generating apparatus according to claim 10 or 16, wherein the model is a neural network. (Configuration 18) A video analysis system including a video analysis device that performs video analysis of a video captured by a camera, and a learning device that learns a model used in the video analysis, The learning device includes: An acquisition means for acquiring related information related to the use conditions of the video analysis device based on the use conditions of the video analysis device; a description generating means for generating description data representing an image that is expected to be a target of video analysis by the video analysis device based on the related information; an image generating means for generating an analysis image based on the description data; a learning means for learning a model used in the video analysis using the analysis image generated by the image generating means; having The video analysis device includes: an analysis means for performing the video analysis using the model trained by the learning means; A video analysis system comprising: (method) An acquisition step of acquiring related information related to the use conditions based on the use conditions of the video analysis; a description generation step of generating description data representing an image that is expected to be a target of the video analysis when the video analysis is used based on the related information; an image generation step of generating an analysis image of a model to be used in the video analysis based on the description data; 13. An image generating method comprising: (program) A computer of the image generating device, An acquisition means for acquiring related information related to a use condition based on the use condition of the video analysis; a description generating means for generating description data representing an image that is expected to be a target of the video analysis when the video analysis is performed based on the related information; an image generating means for generating an analysis image of a model used in the video analysis based on the description data; A program that functions as a [Explanation of symbols]

[0065] 102: In-house mail server, 103: Sales data management server, 104: Video analysis device, 106: User site, 107: News site, 108: Information site, 111: Learning device, 112: Internet

Claims

1. An acquisition means for acquiring related information related to a use condition based on the use condition of the video analysis; a description generating means for generating description data representing an image that is expected to be a target of the video analysis when the video analysis is performed based on the related information; an image generating means for generating an analysis image to be used in the video analysis based on the description data; An image generating device comprising:

2. The image generating device described in claim 1, characterized in that the acquisition means acquires at least one of the following information as the usage conditions: the name of the store in which the camera that captures the image to be analyzed is installed, the type of business in which the camera is used, and information regarding the location in which the camera is installed, and acquires the related information based on the usage conditions.

3. The image generating device of claim 1, characterized in that the acquisition means acquires at least one of the usage conditions, such as the type of camera that captures the image to be analyzed, the shooting resolution of the camera, the shooting time of the camera, and the type of image analysis, and acquires the related information based on the usage conditions.

4. The image generating device according to claim 1, characterized in that the description generation means obtains item information, which is information corresponding to a specified description item, from the related information, and generates the description data written in a format corresponding to the function of the video analysis based on the item information.

5. 5. The image generating apparatus according to claim 4, wherein the predetermined description items are items related to at least one of an object in the background of the image, a lighting condition, a photographing range, and an image quality.

6. 5. The image generating apparatus according to claim 4, wherein the predetermined description items are items that represent attributes of a subject to be analyzed.

7. 2. The image generating device according to claim 1, wherein the acquiring means searches a website for a web page related to the usage conditions, and acquires the related information from text information included in the searched web page.

8. The image generating device according to claim 7, characterized in that the image generating means generates the analysis image based on the description data and features extracted from existing images contained in the searched Web pages.

9. The image generating device according to claim 1 , wherein the acquisition means acquires the related information from a server managed by a user who uses the video analysis.

10. 2. The image generating apparatus according to claim 1, further comprising a learning unit that learns a model used in the video analysis by using the analysis image generated by the image generating unit.

11. 5. The image generating apparatus according to claim 4, wherein the description generating means increases the variation of the description data by detailing the item information.

12. 5. The image generating apparatus according to claim 4, wherein said description generating means controls so as to adjust a ratio of said description data including said specific item information to the whole of said generated description data.

13. 2. The image generating apparatus according to claim 1, wherein said description generating means adjusts said description data generated from said related information in accordance with the importance of said related information.

14. a presentation means for presenting the analysis image generated by the image generation means to a user who uses the video analysis; a receiving means for receiving feedback on the presented analysis image; a correction means for correcting the description data based on the feedback; 2. The image generating apparatus of claim 1, further comprising:

15. 15. The image generating device according to claim 14, wherein the presenting means presents a screen on which the image for analysis and an input field for a comment on the image for analysis are displayed.

16. 11. The image generating apparatus according to claim 10, further comprising a transmitting unit that transmits the model learned by the learning unit to a device that performs the video analysis in the usage environment.

17. The image generating apparatus of claim 10 , wherein the model comprises a neural network.

18. A video analysis system including a video analysis device that performs video analysis of a video captured by a camera, and a learning device that learns a model used in the video analysis, The learning device includes: An acquisition means for acquiring related information related to the use conditions of the video analysis device based on the use conditions of the video analysis device; a description generating means for generating description data representing an image that is expected to be a target of video analysis by the video analysis device based on the related information; an image generating means for generating an analysis image based on the description data; a learning means for learning the model using the analysis image generated by the image generating means; having The video analysis device includes: an analysis means for performing the video analysis using the model trained by the learning means; A video analysis system comprising:

19. An acquisition step of acquiring related information related to the use conditions based on the use conditions of the video analysis; a description generation step of generating description data representing an image that is expected to be a target of the video analysis when the video analysis is used based on the related information; an image generating step of generating an analysis image to be used in the video analysis based on the description data; 13. An image generating method comprising:

20. A computer of the image generating device, An acquisition means for acquiring related information related to a use condition based on the use condition of the video analysis; a description generating means for generating description data representing an image that is expected to be a target of the video analysis when the video analysis is performed based on the related information; an image generating means for generating an analysis image to be used in the video analysis based on the description data; A program that functions as a

Images