Scoring system, method, and program

Abstract

Techniques are provided for scoring images. [Solution] The system includes a reception unit (121) that receives an image including at least one person as a received image, and an evaluation score output unit (122) that acquires an image including one or more people as a learning image and outputs an evaluation score for the received image using an image scoring model (M1) that has been trained to output an evaluation score for the learning image when focusing on a specific person included in the learning image.

Description

[Technical field]

[0001] The present disclosure relates to image scoring techniques. [Background technology]

[0002] In nurseries and schools, many children are photographed and daily reports, albums, etc. are created. For example, Patent Literature 1 discloses an image candidate determination device that assists in selecting images to make the number of images in which each person appears as uniform as possible among images to be made public. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2023-001178 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, Patent Document 1 merely discloses a technique for scoring based on the number of people included in an image, facial expressions, etc., and the scoring of each person is merely based on the individual elements of each person (each person's facial expression, face size, eye opening, etc.). Therefore, there is room for consideration regarding scoring based on information on the entire image, including elements other than the individual elements of each person.

[0005] The non-limiting embodiments of the present disclosure have been made in consideration of the above background, and contribute to providing a technique for more appropriately scoring images that include people. [Means for solving the problem]

[0006] A scoring system according to one embodiment of the present disclosure includes a reception unit that receives an image including at least one person as a received image, and an evaluation score output unit that acquires an image including one or more people as a training image and outputs an evaluation score for the received image using an image scoring model that has been trained to output an evaluation score for the training image when focusing on a specific person included in the training image.

[0007] A method according to one aspect of the present disclosure includes a receiving step of receiving an image containing at least one person as a received image using a scoring system, and an evaluation score output step of acquiring an image containing one or more people as a training image and outputting an evaluation score for the received image using an artificial intelligence model trained to output an evaluation score for the training image when focusing on a specific person included in the training image.

[0008] A computer program according to one aspect of the present disclosure is a program for causing a computer to function as the scoring system described above, and causes the computer to function as each of the above-mentioned parts.

[0009] In addition, these comprehensive or specific aspects may be realized by a system, a method, an integrated circuit, a computer program, or a recording medium, or may be realized by any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium. Effect of the Invention

[0010] According to one aspect of the present disclosure, a technique for more appropriately scoring images that include people can be provided.

[0011] Further advantages and benefits of certain aspects of the present disclosure will become apparent from the specification and drawings, in which such advantages and / or benefits are provided by some of the embodiments and features described in the specification and drawings, respectively, but not necessarily all of which are provided to obtain one or more identical features. [Brief description of the drawings]

[0012] [Figure 1] FIG. 2 is a block diagram showing an example of the functional configuration of the scoring system in the first embodiment. [Diagram 2] FIG. 13 is a diagram showing an example of an image for learning. [Diagram 3] FIG. 13 is a diagram showing an example of an image for learning. [Figure 4] FIG. 13 is a diagram showing an example of an image for learning. [Diagram 5] FIG. 13 is a diagram showing an example of an image for learning. [Figure 6] 4 is a flowchart illustrating an example of the operation of the scoring system in the first embodiment. [Figure 7] FIG. 11 is a block diagram showing an example of the functional configuration of a scoring system in embodiment 2. [Figure 8] 11 is a flowchart illustrating an example of the operation of the scoring system in the second embodiment. [Figure 9] 11 is a flowchart illustrating an example of the operation of the scoring system in the second embodiment. [Figure 10] FIG. 2 is a block diagram illustrating the physical configuration of each computer that makes up the scoring system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, an embodiment of the present disclosure will be described in detail with reference to the drawings as appropriate. However, more detailed description than necessary may be omitted. For example, detailed description of already well-known matters or duplicate description of substantially the same configuration may be omitted. This is to avoid the following description becoming unnecessarily redundant and to facilitate understanding by those skilled in the art. Note that the accompanying drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter described in the claims.

[0014] [Embodiment 1] <<Overview of Scoring System 1>> First, an overview of the scoring system 1 according to the present embodiment will be described. The scoring system 1 according to the present embodiment is an information processing system that scores an evaluation score of an image. As an example, the scoring system 1 can be suitably introduced in facilities such as nursery schools and kindergartens. The scoring system 1 may also be used by users such as nursery teachers and teachers working in facilities such as nursery schools and kindergartens. However, the scoring system 1 may also be used in various schools, nursing homes, hospitals, commercial facilities, and the like other than nursery schools and kindergartens, and the application scene is not particularly limited.

[0015] For example, a facility such as a nursery school or kindergarten may be configured to have one or more cameras installed, and images captured by the cameras may be transmitted to a server, and then transmitted from the server to a terminal device (e.g., a smartphone or a personal computer) owned by a guardian of the child or children. The camera installed in the facility may automatically capture images of the facility repeatedly at a predetermined interval, and the captured images include images of the children in the facility. The camera is connected to a network and can transmit the captured images to the server.

[0016] The server receives images from the camera via the network and stores and accumulates the received images in a storage device. The server may also transmit the accumulated images to terminal devices such as smartphones and personal computers associated with the people involved in the facility (parents of the children, daycare workers, etc.). This allows, for example, parents who leave their children at a facility such as a daycare center or kindergarten to view on their own terminal devices how their children are doing at the facility using the images sent to them.

[0017] Also, for example, a childcare worker at a facility can obtain images sent from a camera installed at the facility and create a photo album showing the children spending time at the facility and post the images, without having to take photos of the children themselves. Note that the childcare worker may take photos of the children themselves using a terminal device owned by the childcare worker, send the photos to a server, and the images may be stored in a storage unit, and there is no particular limitation to this.

[0018] For example, cameras installed in a facility may automatically capture thousands or hundreds of thousands of images per day, and it is not easy for users (childcare workers, parents, etc.) to visually review each image in this large number and select appropriate photos for each subject (e.g., children at a nursery school).

[0019] Therefore, the scoring system 1 of this embodiment may be applied, as an example, to an information processing system for easily selecting appropriate photos for each subject from a large number of images taken by cameras installed in a facility, by performing appropriate and reasonable scoring on images, or for supporting the creation of documents using photos (daily reports, albums, etc.).

[0020] <<Structure of Scoring System 1>> Hereinafter, the scoring system 1 according to the present embodiment will be described in detail with reference to the drawings. FIG. 1 is a block diagram showing an example of the functional configuration of the scoring system 1. The scoring system 1 is an information processing system configured by an information processing device. The scoring system 1 may be configured by one device or may be configured by multiple devices. Furthermore, when an information processing device is configured by multiple devices, the devices do not need to be installed in the same space such as the same room, and may be installed in different rooms, different buildings, different regions, etc., and are not particularly limited.

[0021] 1, the scoring system 1 includes a server 10 and a user terminal 20. The server 10 is an example of an information processing device in the present disclosure. The server 10 and the user terminal 20 are communicatively connected via a network N1. The network N1 connecting the server 10 and the user terminal 20 is a wired LAN (Local Area Network), a wireless LAN, the Internet, a public line network, a mobile data communication network, or a combination thereof.

[0022] (Configuration of user terminal 20) 1, user terminal 20 includes a storage unit 21 and a control unit 22. Control unit 22 acquires various information (input information) input by a user of scoring system 1 via an input device. Input by a user via an input device may be simply described as "input by a user."

[0023] Moreover, the control unit 22 presents to the user various information (output information) transmitted from the server 10 in response to the transmission of information input by the user, by displaying it on the display device. Presenting to the user by displaying it on the display device may be simply described as presenting to the user. Moreover, the storage unit 21 stores information transmitted and received between the server 10.

[0024] (Server 10 configuration) The server 10 includes a storage unit 11, a control unit 12, and an artificial intelligence model M. The storage unit 11 stores information to be transmitted and received between the server 10 and the user terminal 20, as well as information to be transmitted and received between the server 10 and other devices communicatively connected via the network N1.

[0025] The control unit 12 is composed of a reception unit 121, an evaluation score output unit 122, a specified information generation unit 123, a person image detection unit 124, a text generation unit 125, an attribute information output unit 126, an image clipping unit 127, a transmission unit 128, a first application processing unit 129, a second application processing unit 130, a third application processing unit 131, a first document generation unit 132, a second document generation unit 133, and a learning unit 134.

[0026] The artificial intelligence model M includes an image scoring model M1, a text generation model M2, and an attribute information output model M3.

[0027] Note that this is merely one example of the functional configuration of the scoring system 1, and it is not necessary for all functional blocks to be included. For example, if there are functions that do not need to be provided depending on user needs, the configuration may include only the required combination of functional blocks.

[0028] In addition, the configuration in which all the functional blocks included in the control unit 12 are arranged in one server 10 is merely an example, and each functional block may be distributed and arranged in multiple information processing devices, and is not particularly limited. Furthermore, each functional block is not limited to a configuration in which it is realized by one integrated program, and may be a configuration in which it is realized by multiple programs, and is not particularly limited.

[0029] In addition, the configuration in which all models included in the artificial intelligence model M are arranged in one server 10 is merely an example, and each model may be distributed and arranged in a plurality of information processing devices, or may be arranged to utilize a function provided by an external device communicably connected via a network N1 or the like, and is not particularly limited. Furthermore, each model may be realized as one function of a single integrated general-purpose artificial intelligence model, or may be realized as an independent trained model for each model, and is not particularly limited.

[0030] <<Image Scoring>> The following describes the functions of the scoring system 1 regarding scoring of images.

[0031] As an example in this embodiment, in the scoring system 1, the reception unit 121 accepts an image including at least one person as a received image, and the evaluation score output unit 122 outputs an evaluation score for the received image using the image scoring model M1.

[0032] (Reception Department 121) For example, a user inputs an image including at least one person to scoring system 1 via user terminal 20. The image input to scoring system 1 may be input as, for example, JPG file data, but may also be image file data in other formats such as PNG or GIF, and the format of the image file is not particularly limited. Reception unit 121 accepts the image input from user terminal 20 as a received image. The received image accepted by reception unit 121 may be stored in memory unit 11.

[0033] Note that the configuration in which the reception unit 121 receives a received image is one example only, in which the reception unit 121 receives an image input by a user via the user terminal 20, and the reception unit 121 may receive an image from another functional block included in the control unit 12, or may receive an image from another information processing device. Furthermore, the reception unit 121 may receive an image input during batch processing that is executed at a predetermined timing on a large amount of images stored in the storage unit 11, and is not limited to a configuration in which the reception unit 121 receives an image input manually by a user.

[0034] (Overview of image scoring model M1) Image scoring model M1 is an artificial intelligence model (AI model) that is trained to obtain an image containing one or more people as a training image and output an evaluation score for the training image when focusing on an image of a specific person contained in the training image.

[0035] Here, an AI model is a model that artificial intelligence creates by learning input data, and for example, it learns rules and relationships in sets of learning data from a huge number of sets of input learning data, and by optimizing parameters within the model, it becomes possible to output an output according to the learning content for the input. In other words, an AI model is a mechanism for determining what kind of output to return for input data.

[0036] A known learning algorithm can be applied as a learning algorithm for training the AI ​​model. For example, deep learning (CNN: Convolutional Neural Network, RNN: Recurrent Neural Network, LSTM: Long short-term memory, GAN: Generative Adversarial Network, etc.) may be used as an algorithm for training the AI ​​model. The AI ​​model may be trained not only by supervised learning, but also by unsupervised learning, semi-supervised learning, etc. The AI ​​learning model does not necessarily have to have a neural network structure, and may be, for example, an SVM (Support Vector Machine) or a decision tree, etc., and is not particularly limited.

[0037] Here is some brief explanation about CNN etc. In recent years, the use of deep learning as an artificial intelligence technology has become more common. Deep learning is an algorithm that deepens the hierarchy of neural networks, which are a type of machine learning technology. Neural networks are algorithms that are modeled on the brain nerves (neurons) of living organisms.

[0038] For example, a neural network has an input layer, a hidden layer, and an output layer. Each layer has nodes, and the nodes may be connected by edges. In this case, the hidden layer may have multiple layers. Machine learning using a neural network with a deep hidden layer is called deep learning.

[0039] In other words, deep learning is a type of machine learning in which a computer automatically extracts features of input data using a neural network. Deep learning can be applied to both supervised and unsupervised learning. The neural network may be trained using a method such as gradient descent, stochastic gradient descent, or backpropagation.

[0040] For example, each layer of a neural network has a function called an activation function, and edges can have weights. The value of each node is calculated from the value of the node in the previous layer. That is, the value of each node is calculated from the node value in the previous layer, the edge weight value, and the activation function. There are various methods for this calculation, but we will not explain the details here.

[0041] CNN has a proven track record in the field of image recognition, for example. In CNN, the hidden layer consists of a convolutional layer and a pooling layer. The convolutional layer performs filtering on nearby nodes in the previous layer. It is like sliding a filter represented by a matrix over the original input image. This allows you to obtain a feature map.

[0042] The pooling layer shrinks the feature map output from the convolution layer to generate a new feature map. This feature map generation process makes it possible to absorb image misalignment. The pooling layer then processes the local features together.

[0043] In other words, the convolution layer and pooling layer reduce the image while maintaining the characteristics of the input image. The difference from simple image compression is that the image is reduced while maintaining the characteristics of the image, in other words, it is an abstraction process.

[0044] In a CNN, each layer passes meaningful data to the next, and as the layers progress the network can learn higher level features: for example, the first layer might detect local features like edges, the next layer might combine them to detect textures, and the next layer might detect more abstract body features like ears.

[0045] CNN automatically learns the parameters for extracting such features. In this way, CNN is able to recognize or classify input images using the abstract image data stored in the network.

[0046] While CNN mainly handles image data (two-dimensional rectangular data), RNN handles variable-length time-series data, such as audio data. In order to handle variable-length data in a neural network, RNN has a network structure in which the values ​​of the hidden layer are input back into the hidden layer.

[0047] However, RNN has problems such as errors disappearing and the amount of calculations becoming huge when using data from a long time ago. Therefore, it can only process data for a short period of time. That is where LSTM came in. LSTM is a learning model that can solve the shortcomings of RNN and learn long-term time series data.

[0048] As mentioned above, in CNN, it can be said that a feature extractor (or classifier) ​​that captures various features is already created. In transfer learning, the feature extractor part already created in CNN (the layer closer to the input side of the neural network) is reused, and only the classifier part (the layer closer to the output side of the neural network) is retrained. This allows additional learning to be performed with less data than normal learning.

[0049] For example, in image recognition, if a neural network has already been trained to recognize dogs and cats, it can be trained to recognize other animals such as monkeys and chimpanzees by reusing the layers that identify primitive elements and features close to the input side and retraining only the layers that identify more abstract elements and features close to the output side and the layers that classify objects. In this way, the amount of data can be compensated for by performing transfer learning in which the layers close to the output side are retrained.

[0050] (Learning Section 134) Scoring system 1 may be configured to automate learning of image scoring model M1 by learning unit 134. For example, learning unit 134 may perform supervised learning of image scoring model M1 using training data.

[0051] The teacher data may be, for example, a set of a learning image including one or more people and an evaluation score for the learning image when focusing on an image of a specific person included in the learning image. Note that the scoring system 1 may be configured to manually train the image scoring model M1, and is not particularly limited.

[0052] Fig. 2 is a diagram showing an example of a learning image. For example, the learning unit 134 may train an image scoring model M1 using a pair of the learning image shown in Fig. 2 and the evaluation score for the learning image as an example of teacher data.

[0053] 2, the learning image P is an image of three children (kindergarteners A, B, and C) and one adult (teacher D). The image scoring model M1 may acquire the image shown in FIG. 2 from the learning unit 134, for example.

[0054] Furthermore, image scoring model M1 acquires, for example, from learning unit 134, together with image P, a numerical value representing an evaluation score for the learning image when focusing on an image of a specific person contained in image P.

[0055] In addition, the image scoring model M1 may be configured not only to be automatically learned by the learning unit 134, but also to have a configuration in which a pair of an image P and the above-mentioned evaluation score is input to the scoring system 1 by a user instead of the learning unit 134, and is not particularly limited.

[0056] Then, the image scoring model M1 will construct a mechanism that determines what evaluation score to output for an input image by repeatedly updating (i.e., repeatedly learning) the parameters within the model to optimize them based on, for example, the rules and relationships between a huge number of image and evaluation score pairs.

[0057] (Evaluation score output unit 122) The evaluation score output unit 122 outputs an evaluation score for the received image using the trained image scoring model M1 described above. The evaluation score may be output as a numerical value in the range of 0 to 1, for example.

[0058] Furthermore, the evaluation score may be evaluated according to other numerical ranges or predetermined subdivided grades such as A rating, B rating, etc., and is not particularly limited. The evaluation score may be used, for example, for classifying, sorting, or selecting images, as described below. Furthermore, the evaluation score may be presented to the user, for example, via a display or speaker of the user terminal 20. Furthermore, the evaluation score may be stored in the storage unit 11 together with information identifying the reception image to be evaluated (for example, a reception image ID, etc.).

[0059] When the evaluation score output unit 122 outputs the evaluation score, information indicating a person to be focused on when outputting the evaluation score among the people included in the received image may be specified in advance by the user. Alternatively, the evaluation score output unit 122 may autonomously specify an image area of ​​a person to be focused on when outputting the evaluation score among the people included in the received image. For example, the evaluation score output unit 122 uses an AI model that detects a person included in an image to specify an image area of ​​a person to be focused on when outputting the evaluation score for the received image, and then outputs the evaluation score for the received image.

[0060] In addition, when the evaluation score output unit 122 outputs an evaluation score for a received image, it may simultaneously output information indicating which person included in the received image the evaluation score was based on (for example, the person's identification ID) and store this in the memory unit 11 in correspondence with the received image ID and its evaluation score.

[0061] For example, the evaluation score output unit 122 may output a child ID that identifies a child that was focused on when outputting the evaluation score among the people included in the image. The evaluation score output unit 122 may, for example, extract facial features of the child included in the image and compare them with facial images of the child registered in the storage unit 11 or features extracted from these images. Then, the evaluation score output unit 122 may search for data of a child registered in the storage unit 11 that matches or is similar to the facial features of the child included in the image, and output the corresponding child ID.

[0062] The evaluation score output unit 122 may extract facial features of a person included in the image, for example, by using a learning model that receives an image of a face as an input and outputs multidimensional vector information as the facial feature. The same learning model may also be used when registering the facial feature of a person in the storage unit 11. The evaluation score output unit 122 may, for example, calculate the distance of a multidimensional vector corresponding to a plurality of feature amounts, and determine that the facial features match or are similar when the distance is equal to or smaller than a threshold and is the smallest.

[0063] In this way, the evaluation score output by the evaluation score output unit 122 is an evaluation score for the received image when focusing on a specific person; in other words, this is an evaluation score grasped from the entire received image including images of parts other than the image of the focused on person, and is a score that represents an evaluation of the focused on person according to the situation in the received image.

[0064] For example, the evaluation score output by the evaluation score output unit 122 will differ depending on the communication situation between the person included in the reception image that has been focused on and people other than the person that has been focused on (for example, talking to a teacher, listening to the teacher, or looking away and not listening to the teacher at all).

[0065] For example, as described below, the evaluation score output unit 122 may output an evaluation score such that a high score is output if a particular person of interest is talking to a teacher, a medium score is output if the person is listening to the teacher, or a low score is output if the person is looking away and not listening to the teacher at all.

[0066] Note that image scoring techniques have existed for some time, but in these methods, the evaluation items listed for the person in the photograph are merely individual elements related to the person themselves, such as whether their eyes are open, whether they are smiling, whether they are a close-up, etc., and in order to evaluate these evaluation items, each element of the image of the person itself is analyzed and a total value is calculated by weighting the scores of each element, etc.

[0067] In contrast, in the scoring system 1 disclosed herein, as described above, an evaluation score for the received image focusing on a specific person is output based on information grasped from the entire image, including not only information about the person themselves, but also people other than that person, or background images (such as being indoors at the nursery, playing outdoors, or participating in an event such as a field trip).

[0068] <<Specifying the person to focus on in image scoring>> Hereinafter, a configuration for specifying an image area of ​​a person of interest in scoring an image by the scoring system 1 will be described.

[0069] (Specified information) In the scoring system 1, the reception unit 121 receives designation information that designates an image area of ​​a person included in a received image. The reception unit 121 may simultaneously receive the designation information and the received image, or may first receive only the received image and then receive the designation information at a different timing, and is not particularly limited.

[0070] The specified information may be, for example, information on the coordinates of four points representing the range of the image of a person included in the reception image, or information representing the range of the image area of ​​a person by the coordinates of the center point of the image area of ​​the person and the vertical and horizontal distances from the center point.

[0071] Alternatively, for example, a rectangular figure surrounding the image area of ​​the person of interest may be added to the received image, and the receiving unit 121 may acquire information on the rectangular figure as the specified information. That is, the receiving unit 121 is not limited to a configuration in which coordinate information is directly received as the specified information, and the specified information is not particularly limited as long as it is information capable of identifying the range of the image area of ​​the person of interest in the received image. In addition, in the case where a figure surrounding the image area of ​​the person is added to the received image, the shape of the figure is not limited to a rectangle, and may be, for example, a curved figure such as an ellipse.

[0072] Regarding the input of the designated information, the receiving unit 121 may be configured to receive information manually specified by the user, or the receiving unit 121 may be configured to receive information generated by, for example, the designated information generating unit 123 described later, and is not particularly limited.

[0073] (Information for study purposes) Here, the image scoring model M1 has been trained to acquire learning designation information that designates an image area of ​​a person included in a learning image and output an evaluation score for the learning image. That is, as described above, the image scoring model M1 learns to output an evaluation score for an image when focusing on a specific person, but during training, it acquires learning designation information that designates the image area of ​​the specific person to be focused on.

[0074] The learning specification information, like the specification information described above, only needs to be information that can identify the range of the image area of ​​the person of interest in the learning image, and may be information that expresses the range of the image in coordinates, or may be information of a figure added to surround the person in the learning image.

[0075] It is preferable that the designation information received by the reception unit 121 is the same type of designation information as the learning designation information used when the image scoring model M1 used to output the evaluation score was trained. For example, in a configuration in which the reception unit 121 receives coordinate information as the designation information, it is preferable to obtain the coordinate information as the learning designation information and use the trained image scoring model M1.

[0076] Also, for example, in a configuration in which the reception unit 121 receives information on a rectangular shape surrounding a person as the specified information, it is preferable to use the image scoring model M1 that has acquired information on a rectangular shape surrounding a person as the learning specified information and has already been trained. However, for example, since the information on the rectangular shape surrounding a person can also be converted into coordinate information, the specified information received by the reception unit 121 does not necessarily have to be the same type of specified information as the learning specified information used when the image scoring model M1 used to output the evaluation score was trained, and is not particularly limited.

[0077] (An example of learning the image scoring model M1) 3, 4, and 5 are diagrams for explaining an example of a learning image. The image scoring model M1 acquires learning images P1, P2, and P3 shown in Fig. 3, 4, and 5, and learns the evaluation score for each image.

[0078] In this embodiment, the evaluation score is expressed, for example, as a numerical value in the range of 0 to 1 (the closer to 1, the higher the evaluation, and the closer to 0, the lower the evaluation), and an evaluation score is given to the person of interest based on the surrounding situation (for example, the communication situation with teachers and friends, etc.) grasped from the entire image. As described above, the evaluation score is not limited to a configuration expressed by a numerical value in the range of 0 to 1.

[0079] In the learning image P1 shown in FIG. 3, the image of child A is surrounded by a dashed rectangular shape, and the image focuses on child A. In image P1, it can be seen that child A is not looking at teacher D, and is not listening to what teacher D is saying. For this reason, image P1 focusing on child A is not a very good scene for child A, and is therefore scored low. In this embodiment, the evaluation score for image P1 is set to "0.3". The image scoring model M1 acquires the evaluation score "0.3" for image P1 as part of the learning data, and updates parameters within the model as described above to learn about scoring the evaluation score.

[0080] In the learning image P2 shown in FIG. 4, the image of child B is surrounded by a dashed rectangular shape, and the image focuses on child B. In image P2, child B is not talking to teacher D, but is facing teacher D and is seen to be listening carefully. For this reason, image P2 focusing on child B can be said to be an image showing a relatively good scene for child B, and therefore is scored as a moderate evaluation. In this embodiment, the evaluation score of image P2 is set to "0.6". The image scoring model M1 acquires the evaluation score "0.6" for image P2 as part of the learning data, and updates parameters inside the model as described above to learn about scoring the evaluation score.

[0081] In the learning image P3 shown in FIG. 5, the image of child C is surrounded by a dashed rectangular shape, and the image focuses on child C. In image P3, child C can be seen talking to teacher D using gestures. For this reason, image P3 focusing on child C can be said to be an image showing a very good scene for child C, and therefore receives a high evaluation score. In this embodiment, the evaluation score for image P3 is set to "0.9". The image scoring model M1 acquires the evaluation score "0.9" for image P3 as part of the learning data, and updates parameters within the model as described above to learn about scoring the evaluation score.

[0082] In addition to the above images P1, P2, and P3, a huge number of images are input to the image scoring model M1 as learning data, and evaluation scores for each image when focusing on a specific person included in those images are also input. Similarly to learning about scoring using the above images P1, P2, and P3, the image scoring model M1 repeats learning using a huge number of images about scoring evaluation scores for images when focusing on a specific person, thereby improving the accuracy of scoring.

[0083] In the example of learning of image scoring model M1 described using Figures 3, 4, and 5, an example was described in which information about rectangular figures surrounding human images contained in the acquired learning images P1, P2, and P3 is acquired as specified information for learning, and learning is performed using that information. However, as described above, for example, in a configuration in which the scoring system 1 accepts coordinate information as specified information and outputs an evaluation score, the configuration may also be such that coordinate information is acquired as specified information for learning and learning is performed.

[0084] Incidentally, the image scoring model M1 may be configured, for example, in a case where a learning image includes a plurality of persons, to acquire automatically generated learning designation information for each person and learn from the acquired information.

[0085] In this case, for example, the learning unit 134 may automatically extract people included in the image using a human image detection AI model or the like, and then automatically generate learning specification information for each person included in the image, focusing on each person in turn.

[0086] Then, the image scoring model M1 acquires the learning specification information as well as the evaluation score when focusing on the person specified by the learning specification information, and performs learning using the set of these.

[0087] In addition, the learning designation information that specifies the image area of ​​a person contained in the learning image may be configured to be automatically generated using a human image detection AI model, as described above, or may be configured to be manually specified by a user, and is not particularly limited.

[0088] In this way, the image scoring model M1 has been trained to acquire the learning specification information as part of the learning data and use this information to output an evaluation score for an image, so that the evaluation score output unit 122 can use this image scoring model M1 to output an evaluation score for the received image based on the specification information received by the reception unit 121.

[0089] (Specified information generation unit 123) In the image scoring model M1, the designation information generating unit 123 may generate designation information for each person in sequence for image regions of multiple people included in the received image.

[0090] The designated information generating unit 123 can detect a human image included in the received image by using, for example, a human image detection AI model, etc. The designated information generating unit 123 may be configured to detect a human image by using, for example, the function of the human image detecting unit 124 described later, or may have a function similar to that of the human image detecting unit 124 itself.

[0091] In addition, various image recognition technologies may be used to detect human images, including deep learning, SVM, decision tree, k-nearest neighbor algorithm, Gaussian mixture model, Boston matching, computer vision, etc., and are not particularly limited.

[0092] The human image detection AI model is trained in advance to, for example, receive image data as input and output an image area in which a person included in the image is captured as an image surrounded by coordinate information or a rectangular shape. The human image detection AI model is trained using training data in which image data is associated with data indicating an image area in which a person included in the image is captured. In addition, when the person to be detected based on an image taken at a facility such as a nursery school or kindergarten is a child, it is expected that the accuracy of detecting children can be improved by creating training data using an image in which a child is captured.

[0093] Then, the designation information generating unit 123 generates designation information for each of the detected persons. For example, the designation information generating unit 123 may generate, as the designation information, coordinate information of four points that indicate the range of an image area of ​​the person included in the received image.

[0094] Alternatively, the designation information generating unit 123 may be configured to add a rectangular figure surrounding the image of a person to the received image. As described above about the designation information, information on the rectangular figure surrounding a person is one type of designation information, and this configuration is also included in the configuration for generating the designation information.

[0095] In this way, the designated information generation unit 123 automatically extracts the image area of ​​the person included in the reception image, and automatically generates designated information for each person included in the reception image, focusing on them one by one in order, thereby automatically outputting evaluation scores in succession when focusing on a specific person in the reception image.

[0096] This significantly improves convenience by eliminating the need for users to manually input specific information for each image individually when assigning evaluation scores to images that contain a large number of people, which are taken in large numbers every day, with a focus on a specific person.

[0097] (Person image detection unit 124) In the scoring system 1, the human image detection unit 124 may extract a feature amount of the received image to detect an image of a person. Designation information specifying an image area of ​​the detected person may be generated by the human image detection unit 124, or may be generated by the above-mentioned designation information generation unit 123. Then, the reception unit 121 receives designation information specifying an image area of ​​the person detected by the human image detection unit 124.

[0098] For example, the human image detection unit 124 detects images of people using a human image detection AI model equipped with a feature extractor. This human image detection AI model is a model that has been trained to detect human images.

[0099] The feature extractor extracts features that characterize the human image from the image. The features are represented, for example, by a multidimensional feature vector. Furthermore, the human image detection AI model may include a classifier and classify the human image according to the feature vector extracted by the feature extractor.

[0100] For example, a feature extractor constituting the human image detection unit 124 converts an image of a person into a multidimensional vector with a label that identifies the person (or the person's attributes, etc.). A large number of images containing people are input to the feature extractor as learning data, and the image of the person is converted into a multidimensional vector with a label that identifies the person or the person's attributes, etc., and placed in a multidimensional vector space. Furthermore, a human image detection AI model may be constructed by repeatedly training the classifier to create a boundary surface that allows the classifier to identify each person in the multidimensional vector space.

[0101] It should be noted that the human image detection unit 124 is not limited to the configuration described above, and may detect a human image based on, for example, the distance of a feature vector, or may be configured using other image recognition technology based on features, and is not particularly limited.

[0102] <<Classify / sort / select images based on scores>> (First application processing unit 129) In the scoring system 1, the first application processing unit 129 may perform at least one of classification, sorting, and selection processes on the received images based on predetermined conditions related to the evaluation score output by the evaluation score output unit 122.

[0103] For example, if a specified condition is set such that the received images are classified into three types, namely high rating (evaluation score greater than 0.7), medium rating (0.3 to 0.7), and low rating (less than 0.3), depending on the evaluation score when focusing on child A, the first application processing unit 129 performs classification processing of the received images so as to meet this condition.

[0104] Also, for example, if a specified condition is set such that the received images are sorted in order of the highest evaluation score when focusing on child A, the first application processing unit 129 performs a sorting process on the received images so as to meet this condition.

[0105] Also, for example, if a predetermined condition is set such that, among the evaluation scores for the received images when focusing on child A, those with high evaluation scores (e.g., 0.7 or higher) are selected, the first application processing unit 129 performs a selection process for received images that meet this condition.

[0106] This allows the user to check the list of image files sorted in descending order of rating, and therefore allows the user to easily find a desired image file, for example, when preferentially sending highly rated images to the parents of children in the kindergarten. Note that the predetermined condition is not particularly limited and may be set as a combination of multiple conditions, such as a combination of a sorting condition and a selection condition.

[0107] The evaluation score may be stored in the storage unit 11 in association with the received image (or an image ID identifying the received image) and information identifying the person focused on when the evaluation score is output (person identification ID). In this case, a different image identification ID may be set for the same received image according to the person identification ID. This allows the first application processing unit 129 to classify, sort, or select the received images stored in the storage unit 11 according to a predetermined condition related to the evaluation score.

[0108] <<Text generation>> (Text Generation Unit 125) In the scoring system 1, the text generating unit 125 may generate text explaining the content of the image included in the received image. For example, the text generating unit 125 generates text explaining the content of the image included in the received image by using a text generation model M2 that has been trained to generate text explaining the content of the image when focusing on a specific person included in the learning image.

[0109] For example, if a reception image contains images of child A and child B, child A is shown facing away from the teacher and talking to a friend, while child B is shown facing the teacher and listening to him / her.

[0110] In this case, the text generating unit 125 may generate, for example, text such as "Kindergartener A is talking with his friend and not listening to the teacher" as text explaining the content of the image when focusing on child A. The text generating unit 125 may also generate, for example, text such as "Kindergartener B is listening carefully to the teacher" as text explaining the content of the image when focusing on child B.

[0111] The text generation model M2 is trained in advance to, for example, receive image data as an input, and when an image area in which a person included in the image is captured is designated by learning designation information or the like, output text information that describes the content of the entire image when the person is focused on. The text generation model may be trained using a large amount of learning data in which image data and learning designation information are associated with text expressing the content.

[0112] The text generation model may be configured as, for example, a visual language model (VLM). A visual language model is a type of artificial intelligence, similar to a large-scale language model. A large-scale language model can statistically learn the probability distribution of words and sentences from a huge amount of text data and generate natural language output like a human for a given input (such as a prompt).

[0113] Large-scale language models are also a type of deep learning model, and are based on the aforementioned neural network, a computational model that mimics the function of neurons in the human brain. A neural network has a multi-layered structure that receives a huge amount of input data and adjusts its parameters by learning patterns.

[0114] A large-scale language model is composed of a huge neural network with a huge number of parameters, and is trained using a large amount of text data. As a result, the large-scale language model has internal linguistic knowledge such as grammar, meaning, and context, and can generate appropriate output for a given input.

[0115] Like large-scale language models, visual language models are pre-trained using pairs of massive amounts of text data and image data, allowing them to generate appropriate text output that describes the content of a given image input.

[0116] (Second application processing unit 130) In the scoring system 1, the second application processing unit 130 may perform at least one of classification, sorting, and selection processes on the received image based on predetermined conditions related to at least one of the evaluation score output by the evaluation score output unit 122 and the text generated by the text generation unit 125.

[0117] The classification, sorting, and selection based on the evaluation score are the same functions as those of the first application processing unit 129 described above, and the description will be omitted. The second application processing unit can further perform processing according to conditions related to the text generated by the text generating unit 125. For example, the second application processing unit 130 may select a received image for which text containing positive words such as "smile," "fun," and "cheerful" has been generated. The second application processing unit 130 may also classify the received images into those for which text containing negative words has been generated and those for which text containing positive words has been generated. Furthermore, the second application processing unit 130 may sort the received images in alphabetical order, for example, based on the generated text or main words contained in the text.

[0118] This allows the user to easily extract a set of image files in which text containing positive words has been generated (i.e., image files with high priority as candidates for inclusion in daily reports to be reported to parents, graduation albums, etc.) The predetermined condition is not particularly limited and may be set as a combination of multiple conditions, such as a combination of a condition related to the text and a condition related to the evaluation score.

[0119] The evaluation score may be stored in the storage unit 11 in association with the received image (or an image ID identifying the received image), information identifying a person focused on when outputting the evaluation score (person identification ID), and the text generated by the text generation unit 125. In this case, a different image identification ID may be set for the same received image according to the person identification ID. This allows the second application processing unit 130 to classify, sort, or select the received images stored in the storage unit 11 according to predetermined conditions related to the evaluation score and text.

[0120] <<Output of attribute value>> (Attribute information output unit 126) In the scoring system 1, the attribute information output unit 126 may output attribute information of a person in an image that is focused on when outputting an evaluation score. The attribute information output unit 126 outputs attribute information using an attribute information output model M3 that has been trained to output attribute information related to attributes of a person included in an image. In addition, the attribute information output unit 126 may, for example, use designation information to extract an image of a person included in a received image, and output the attribute information of the person using the attribute information output model M3.

[0121] The attribute information output model M3 may be configured as an AI model that, when an image of a person is input, outputs various attributes of the person (e.g., gender, hair length, length and color of each part of clothing, type of dress, whether or not a hat is worn, whether or not a backpack or bag is worn, facial expression, etc.) as attribute information, or it may be configured as an AI model that outputs the degree of certainty (e.g., an index value that expresses the degree of certainty numerically) about the various attributes of the person as attribute information.

[0122] That is, the attribute information output model M3 can be pre-trained to accept, for example, an image of a person as input and output information representing the attributes themselves or the degree of certainty regarding various attributes of the person depicted in the image (e.g., gender, hair length, length and color of each part of clothing, type of dress, whether or not the person is wearing a hat, whether or not the person is wearing a backpack or bag, etc.).

[0123] The attribute information output model M3 may be trained using learning data (a set of input and output data) that associates, for example, an image of a person with label information that indicates the correct answer for various attributes of the person depicted in the image (for example, "1" if the attribute item applies, and "0" if the attribute item does not apply).

[0124] (Third application processing unit 131) In the scoring system 1, the third application processing unit 131 may perform at least one of classification, sorting, and selection processes on the received image based on predetermined conditions related to at least one of the evaluation score output by the evaluation score output unit 122 and the attribute information output by the attribute information output unit 126.

[0125] The classification, sorting, and selection based on the evaluation score are functions similar to those of the above-mentioned first application processing unit 129, and therefore a description thereof will be omitted. The third application processing unit can further perform processing according to conditions related to the attribute information output by the attribute information output unit 126. For example, the third application processing unit 131 may select, classify, or sort the received images by attributes of male or female. Alternatively, the third application processing unit 131 may select, classify, or sort the received images by attributes of smiling or crying faces.

[0126] This allows the user to easily extract, for example, a set of desired image files (i.e., image files with high priority as candidates for inclusion in daily reports to be reported to parents, graduation albums, etc.) Note that the predetermined condition is not particularly limited and may be set as a combination of multiple conditions, such as a combination of a condition related to an attribute and a condition related to an evaluation score.

[0127] The evaluation score may be stored in the storage unit 11 in association with the received image (or an image ID identifying the received image), information identifying a person focused on when outputting the evaluation score (person identification ID), and the attribute information output by the attribute information output unit 126. In this case, a different image identification ID may be set for the same received image according to the person identification ID. This allows the third application processing unit 131 to classify, sort, or select the received images stored in the storage unit 11 according to predetermined conditions related to the evaluation score and the attribute information.

[0128] <<Document creation>> (First document generation unit 132) In the scoring system 1, the first document generation unit 132 may generate a predetermined document using a reception image that has been subjected to at least one of selection, classification, and sorting processing by the first application processing unit 129, the second application processing unit 130, or the third application processing unit 131. The predetermined document may be, for example, a daily report or a graduation album that is sent daily to the guardians of the children, or document data that is a draft for creating them.

[0129] The first document generating unit 132 may generate a document by, for example, selecting reception images with high evaluation scores (e.g., scores of 0.8 or more) for each child and inserting photos of the selected reception images into a predetermined format such as a daily report. Furthermore, the first document generating unit 132 may generate a document by using text generated by the text generating unit 125 and attribute information output by the attribute information output unit 126 in addition to the selected, classified, or sorted reception images.

[0130] Furthermore, the first document generation unit 132 may generate a document using a large-scale language model. In this case, for example, the first document generation unit 132 may generate a prompt according to a predetermined format based on the text generated by the text generation unit 125 and the attribute information output by the attribute information output unit 126, and input the prompt to the large-scale language model to cause the large-scale language model to generate a document such as a daily report.

[0131] <<Image cropping>> (Image Cutout 127) In the scoring system 1, the image cut-out unit 127 may cut out an image of the person that has been focused on when the evaluation score is output, based on the text generated by the text generation unit 125 and the evaluation score.

[0132] As described above, the text generating unit 125 generates text explaining the contents of the image included in the received image, and since this text allows the reason for a high or low evaluation score to be understood, the image clipping unit 127 may clip a range of the image according to the content of the text. For example, if text with the content "having fun playing with friends" is generated for an image with a high evaluation score, the image clipping unit 127 may clip a range of the image that allows the appearance of playing with friends to be understood.

[0133] In this case, the image cropping unit 127 can be configured, for example, as an image cropping AI model that has been pre-trained to accept as input an image including a person, information specifying the image area of ​​the person in focus, and text information about the person, and to crop and output an image of the person in focus within a range that allows the content of the text information to be grasped.

[0134] For example, when an image containing a person, information specifying the image area of ​​the person, and text information such as "playing with friends" are input into an image cropping AI model, an image range that allows the user to understand that the person is playing with friends (for example, an image range including the friends who are playing together) is cropped and output.

[0135] For example, an image cropping AI model may be trained using training data (a set of input and output data) that corresponds image data, information specifying the image area of ​​a person of interest, and text information, together with an image in which the person has been included in the image and a range within the image that allows the content of the text to be understood.

[0136] (Second document generation unit 133) In the scoring system 1, the second document generating unit 133 may generate a document using an image of a person cut out by the image cutting unit 127. The document to be generated may be, for example, a daily report or a graduation album that is sent to the guardians of the children every day, or document data that is a draft for creating them.

[0137] The image (cut-out image) cut out by the image cut-out unit 127 is stored in the memory unit 11 in association with the text generated by the text generation unit 125, and the second document generation unit 133 may generate a document, for example, by selecting a cut-out image that matches text information specified by the user and inserting the cut-out image into a predetermined format such as a daily report.

[0138] For example, when a user creates a daily report for a kindergartener, the user inputs information to the scoring system 1 instructing it to generate a draft of the daily report using an image in which the text "playing with friends" is generated. The second document generation unit 133 then reads out a cut-out image corresponding to the text "playing with friends" from the memory unit 11 and generates a document such as a daily report with a photograph using the read cut-out image.

[0139] (Transmitter 128) The transmission unit 128 may be configured as, for example, a messaging app. That is, the server 10 is connected to the Internet via the network N1, and the transmission unit 10 can perform data communication with a mobile terminal owned by the guardian by using the function of the messaging app.

[0140] In the scoring system 1, the image of the person cut out by the image cutout unit 127 may be transmitted to an information processing terminal owned by a related person of the person. For example, the transmission unit 128 may transmit the image to a mobile terminal or the like owned by a related person of the child appearing in the cut-out image, such as a parent of the child or a parent of a friend appearing in the image.

[0141] <<An example of the process flow in scoring system 1>> An example of the operation of the scoring system 1 configured as above will be described with reference to Fig. 6. Fig. 6 is a flowchart for explaining an example of the operation of the scoring system 1. Note that the flowchart shown in Fig. 6 is merely an example of the flow of processing, and is not particularly limited, and may include other steps, or the same steps may be executed repeatedly, or the order of steps may be changed or some steps may not be executed, depending on a user request.

[0142] In step S101, as an example, the control unit 22 of the user terminal 20 transmits an instruction to input an image from the user to the server 10. Note that the image may be uploaded from the user terminal 20 to the server 10, or a large number of images stored in the storage unit 11 may be input by batch processing or the like. For example, in the case of input by batch processing, when the user performs an operation to instruct execution of batch processing via the user terminal 20, an instruction to execute the batch processing is transmitted to the server 10.

[0143] In step S102, as an example, the receiving unit 121 of the server 10 receives an image including a person as a received image in response to an image input instruction from the user terminal 20. Note that, although the details of the designation information are as described above, the designation information may be configured to separately designate coordinate information, or may be included in the received image as a rectangular figure surrounding the person.

[0144] In step S103, as an example, the evaluation score output unit 122 of the server 10 outputs an evaluation score when focusing on a specific person for the received image received by the reception unit 121. The evaluation score output unit 122 transmits the output evaluation score to the user terminal 20. For example, the evaluation score output unit 122 may output an evaluation score for each image when focusing on all children appearing in each image individually, and store it in the storage unit 11 together with the child ID and the image ID.

[0145] In step S104, for example, the control unit 22 of the user terminal 20 presents the evaluation score for the received accepted image to the user. The user can confirm the presented evaluation score.

[0146] In step S105, as an example, the control unit 22 of the user terminal 20 receives an instruction to select an accepted image from the user, and transmits information indicating the content of the instruction to the server 10. For example, if the evaluation score range is set to three levels, A (high evaluation), B (medium evaluation), and C (low evaluation), an instruction to select images in the evaluation score range A for all the children is transmitted from the user.

[0147] In step S106, for example, in the server 10, the first application processing unit 129 selects a received image that meets the conditions specified by the user regarding the evaluation score. The second application processing unit 130 may select a received image that meets the conditions specified by the user regarding text information indicating the image content in addition to the conditions regarding the evaluation score. The third application processing unit 131 may select a received image that meets the conditions specified by the user regarding the person's attribute information in addition to the conditions regarding the evaluation score.

[0148] In step S107, as an example, the control unit 22 of the user terminal 20 presents the reception image selected by the first application processing unit 129 (or the second application processing unit 130 or the third application processing unit 131) to the user. The user checks the presented reception image.

[0149] In step S108, for example, the control unit 22 of the user terminal 20 transmits to the server 10 information instructing the generation of a draft of a document input by the user (such as a daily report for each child to be sent to parents).

[0150] In step S109, as an example, the first document generation unit 132 (or the second document generation unit 133) of the server 10 generates a draft of a document (such as a daily report for each child to be sent to parents).

[0151] In step S110, for example, the control unit 22 of the user terminal 20 presents a draft of the received document (such as a daily report for each child to be sent to the guardian) to the user. The user checks the presented draft of the document.

[0152] In step S111, for example, the user modifies a draft of the document (such as a daily report to be sent to the guardian) as necessary, and the control unit 22 of the user terminal 20 transmits the document reflecting the modifications input by the user to the server 10. In the server 10, the modified document is stored in the storage unit 11. Thereafter, for example, the transmitting unit 128 may distribute the document to an information terminal such as a smartphone of the guardian at an appropriate time.

[0153] This is the end of the operation of scoring system 1. As described above, the flowchart shown in Figure 6 merely explains one example of the operation of scoring system 1, and the operation of scoring system 1 is not limited thereto.

[0154] [Embodiment 2] The configuration described in embodiment 1 is merely an example, and the server side does not necessarily have to have all of the above-mentioned functional blocks; for example, the client side may have a configuration that includes all of the above-mentioned functional blocks.

[0155] Also, for example, the server side may be equipped with some of the above-mentioned functional blocks, and the client side may be equipped with the remaining functional blocks, and the arrangement of the functional blocks on the server side and the client side may be combined in any manner according to the specifications required in each use case, and is not particularly limited.

[0156] The scoring system 1A according to this embodiment will be described in more detail below. For ease of explanation, the same reference numerals are given to components having the same functions as those described in the above embodiment, and the description thereof will not be repeated.

[0157] <<Structure of Scoring System 1A>> Fig. 7 is a block diagram showing an example of a functional configuration of the scoring system 1A. As shown in Fig. 7, the scoring system 1A differs from the scoring system 1 according to the first embodiment in that it includes a user terminal 20A instead of the user terminal 20 and a server 10A instead of the server 10.

[0158] Server 10A differs from server 10 in that it includes a control unit 12A instead of control unit 12, a storage unit 11A instead of storage unit 11, and further in that it does not include artificial intelligence model M.

[0159] User terminal 20A differs from user terminal 20 in that it includes a storage unit 21A instead of storage unit 21, a control unit 22A instead of control unit 22, and further includes an artificial intelligence model M' and an imaging unit 23A.

[0160] The control unit 22A is composed of a reception unit 221, an evaluation score output unit 222, a specified information generation unit 223, a person image detection unit 224, a text generation unit 225, an attribute information output unit 226, an image clipping unit 227, a transmission unit 228, a first application processing unit 229, a second application processing unit 230, a third application processing unit 231, a first document generation unit 232, a second document generation unit 233, a learning unit 234, an alert unit 235, and a storage processing unit 236.

[0161] The artificial intelligence model M' includes an image scoring model M1', a text generation model M2', and an attribute information output model M3'.

[0162] Here, the reception unit 221 is the reception unit 121, the evaluation score output unit 222 is the evaluation score output unit 122, the designation information generation unit 223 is the designation information generation unit 123, the human image detection unit 224 is the human image detection unit 124, the text generation unit 225 is the text generation unit 125, the attribute information output unit 226 is the attribute information output unit 126, the image clipping unit 227 is the image clipping unit 127, the transmission unit 228 is the transmission unit 128, the first application processing unit 229 is the first application processing unit 129, the second application processing unit 230 is the second application processing unit The application processing unit 130 has similar functions, the third application processing unit 231 has similar functions to the third application processing unit 131, the first document generation unit 232 has similar functions to the first document generation unit 132, the second document generation unit 233 has similar functions to the second document generation unit 133, the learning unit 234 has similar functions to the learning unit 134, the image scoring model M1' has similar functions to the image scoring model M1, the text generation model M2' has similar functions to the text generation model M2, and the attribute information output model M3' has similar functions to the attribute information output model M3.

[0163] That is, in this embodiment, each functional block included in the server-side control unit 12 in the first embodiment is included in the client-side control unit 22A, but the only difference is that the information processing of each functional block is executed in the client-side information processing device, and the functions of each functional block are the same as those described in the first embodiment. The alert unit 235 and the save processing unit 236 will be described later.

[0164] <<An example of the process flow in scoring system 1A>> In this embodiment, the user terminal 20A is equipped with an imaging unit 23A, and an example of a case in which a user (e.g., a nursery school teacher) uses the user terminal 20A to take pictures of children at the nursery school will be described with reference to Figures 8 and 9.

[0165] Note that the flowcharts shown in Figures 8 and 9 are merely one example of a processing flow, and are not particularly limited, and for example, depending on a user request, other steps may be included, the same steps may be executed repeatedly, the order of the steps may be changed, or some steps may not be executed.

[0166] 8 is a flowchart for explaining an example of the operation of scoring system 1 A. In step S201, as an example, imaging unit 23A of user terminal 20A captures an image including a person (for example, an image showing a child at a nursery school) in response to a user operation, and the image is input to scoring system 1 A.

[0167] In step S202, as an example, the reception unit 221 of the user terminal 20A receives an image including a person captured by the imaging unit 23A as a received image. Note that the designation information is as described in the above embodiment, but may be generated by the designation information generation unit 223 and included in the received image as a rectangular figure surrounding the person, for example.

[0168] In step S203, as an example, the evaluation score output unit 222 of the user terminal 20A outputs an evaluation score when focusing on a specific person for the received image received by the reception unit 221. For example, the evaluation score output unit 222 may output an evaluation score for each captured image when focusing on all children appearing in each image individually, and store the evaluation scores in the storage unit 21A together with the child ID and the image ID.

[0169] In step S204, for example, the control unit 22A of the user terminal 20A presents the evaluation score for the received image to the user. The user can check the presented evaluation score.

[0170] At this time, the alert unit 235 may present an alert to the user, for example, if the evaluation score is lower than a predetermined score. That is, an evaluation score (alert standard score) that is a standard for presenting a predetermined alert is stored in the storage unit 21A, and the alert unit 235 determines whether the evaluation score output by the evaluation score output unit 222 is lower than the alert standard score, and if lower, outputs an alert to the effect that the evaluation score is lower than the alert standard score. In response to the presentation of this alert, the user can then retake the photograph of the child in question.

[0171] FIG. 9 is a flowchart illustrating an example of the operation of the scoring system 1A. Steps S301, S302, and S303 are similar processing steps to steps S201, S202, and S203 shown in FIG. 8, respectively, and therefore the description thereof will be omitted.

[0172] In step S304, for example, the control unit 22A of the user terminal 20A presents the evaluation score for the received image to the user. The user can check the presented evaluation score.

[0173] In step S305, as an example, the control unit 22A of the user terminal 20A receives an instruction to select an accepted image from the user. For example, if the range of the evaluation score is set to three levels, A (high evaluation), B (medium evaluation), and C (low evaluation), the user inputs an instruction to select an image with an evaluation score range of A for all the children.

[0174] In step S306, for example, in the user terminal 20A, the first application processing unit 229 selects a received image that meets the conditions specified by the user regarding the evaluation score. The second application processing unit 230 may select a received image that meets the conditions specified by the user regarding the text information indicating the image content in addition to the conditions regarding the evaluation score. The third application processing unit 231 may select a received image that meets the conditions specified by the user regarding the person's attribute information in addition to the conditions regarding the evaluation score.

[0175] In step S307, as an example, the control unit 22A of the user terminal 20A presents the reception image selected by the first application processing unit 229 (or the second application processing unit 230 or the third application processing unit 231) to the user. The user checks the presented reception image.

[0176] In step S308, for example, the control unit 22A of the user terminal 20A receives an instruction to save the received images input from the user. For example, the user inputs an instruction to save only the received images with an evaluation score of A (high evaluation).

[0177] In step S309, as an example, when the saving processing unit 236 receives an instruction from the user to save only received images with an evaluation score of A (highly rated), it transmits to the server 10A only the received images with an evaluation score of A (highly rated) that correspond to the content of the instruction.

[0178] In step S310, as an example, when the control unit 12A of the server 10A receives the accepted image transmitted from the user terminal 20A by the saving processing unit 236, the control unit 12A stores the accepted image in the memory unit 11A. In this manner, by configuring only accepted images that satisfy a predetermined condition to be transmitted to the server 10A and stored therein, it is possible to conserve the storage capacity of storage such as the memory unit 11A.

[0179] The save processing unit 236 may be configured to save a received image that satisfies a predetermined condition in the storage unit 21A of the user terminal 20A. Alternatively, for example, predetermined conditions for images to be saved may be stored in the storage unit 21A, and the save processing unit 236 may read the conditions for images to be saved from the storage unit 21A, and sequentially store only the received images that meet the conditions in the storage unit 11A or the storage unit 21A, and is not particularly limited.

[0180] [Software implementation example] The control block of the server 10 may be realized by a logic circuit (hardware) formed in an integrated circuit (IC chip) or the like, or may be realized by software. In the latter case, each of the server 10 and the user terminal 20 is configured using, for example, a computer (electronic calculator).

[0181] (Physical configuration of Server 10) 10 is a block diagram illustrating an example of the physical configuration of a computer used as the server 10 and the user terminal 20. Note that the server 10A has the same physical configuration as the server 10, and the user terminal 20A has the same physical configuration as the user terminal 20.

[0182] 10, the server 10 can be configured by a computer including a bus 110, a processor 101, a main memory 102, an auxiliary memory 103, and a communication interface 104. The processor 101, the main memory 102, the auxiliary memory 103, and the communication interface 104 are connected to each other via the bus 110.

[0183] The processor 101 may be, for example, a CPU (Central Processing Unit), a microprocessor, a digital signal processor, a microcontroller, or a combination of these.

[0184] The main memory 102 may be, for example, a semiconductor RAM (random access memory).

[0185] For example, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), or a combination of these may be used as the auxiliary memory 103. A program for causing the processor 101 to execute the above-mentioned operations of the server 10 is stored in the auxiliary memory 103. The processor 101 loads the program stored in the auxiliary memory 103 onto the main memory 102, and executes each instruction included in the loaded program.

[0186] The communication interface 104 is an interface that connects to the network N1.

[0187] In this example, the processor 101 and the communication interface 104 are an example of a hardware element that realizes the control unit 12. The main memory 102 and the auxiliary memory 103 are an example of a hardware element that realizes the storage unit 11.

[0188] (Physical configuration of user terminal 20) 10, the user terminal 20 can be configured by a computer including a bus 210, a processor 201, a main memory 202, an auxiliary memory 203, a communication interface 204, and an input / output interface 205. The processor 201, the main memory 202, the auxiliary memory 203, the communication interface 204, and the input / output interface 205 are connected to one another via the bus 210. An input device 206 and an output device 207 are connected to the input / output interface 205.

[0189] The processor 201 may be, for example, a CPU, a microprocessor, a digital signal processor, a microcontroller, or a combination of these.

[0190] The main memory 202 may be, for example, a semiconductor RAM.

[0191] The auxiliary memory 203 may be, for example, a flash memory, a HDD, an SSD, or a combination of these. A program for operating the computer as the user terminal 20 is stored in the auxiliary memory 203. The processor 201 loads the program stored in the auxiliary memory 203 onto the main memory 202, and executes each instruction included in the loaded program. In addition, the auxiliary memory 203 stores various data referenced by the processor 201 for operating the computer as the user terminal 20.

[0192] The communication interface 204 is an interface for connecting to a network.

[0193] As the input / output interface 205, for example, a USB interface, a short-range communication interface such as infrared or Bluetooth (registered trademark), or a combination of these is used.

[0194] For example, a keyboard, a mouse, a touch pad, a microphone, or a combination of these, etc. is used as the input device 206. For example, a display, a printer, a speaker, or a combination of these, etc. is used as the output device 207.

[0195] In this example, the processor 201 and the communication interface 204 are an example of a hardware element that realizes the control unit 22. The main memory 202 and the auxiliary memory 203 are an example of a hardware element that realizes the storage unit .

[0196] Each of the above-mentioned programs may be recorded in an external recording medium instead of being stored in the auxiliary memories 103 and 203, and may be read from the external recording medium and supplied to the corresponding computer. The external recording medium may be a computer-readable "non-transitory tangible medium," such as a tape, a disk, a card, a semiconductor memory, or a programmable logic circuit. Each of the above-mentioned programs may be supplied to the computer via any transmission medium (such as a communication network or a broadcast wave). One aspect of the present invention may also be realized in the form of a data signal embedded in a carrier wave, in which each program is embodied by electronic transmission.

[0197] The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present invention.

[0198] <<Summary of this Disclosure>> The following mainly describes the operation and effect of the scoring system according to one aspect of the present disclosure. All configurations described below can also be used as configurations of this embodiment.

[0199] A scoring system according to one embodiment of the present disclosure includes a reception unit that receives an image including at least one person as a received image, and an evaluation score output unit that acquires an image including one or more people as a training image and outputs an evaluation score for the received image using an image scoring model that has been trained to output an evaluation score for the training image when focusing on a specific person included in the training image.

[0200] According to the above configuration, an evaluation score for a received image is output when a specific person included in the received image is focused on. This makes it possible to output an evaluation score for the focused person, including the situation of the person as understood from the entire received image (such as listening to a teacher, playing happily with friends, etc.), and therefore allows for appropriate scoring of the image.

[0201] In a scoring system according to one aspect of the present disclosure, it is preferable that the reception unit receives designation information specifying an image area of ​​a person included in the received image, the image scoring model is trained to obtain learning designation information specifying an image area of ​​a person included in the training image and output an evaluation score for the training image, and the evaluation score output unit uses the image scoring model to output the evaluation score for the received image based on the designation information.

[0202] According to the above configuration, the image area of ​​a person to be focused on is specified when outputting the evaluation score, thereby improving the accuracy of the evaluation score output from the image scoring model and enabling more appropriate scoring of images.

[0203] In the scoring system according to one aspect of the present disclosure, it is preferable that the designation information generation unit generates the designation information for each of a plurality of persons included in the reception image in sequence.

[0204] According to the above configuration, specified information can be automatically generated for all people included in the reception image, so that evaluation scores for the reception image can be output without omission when focusing on each person, making it easy to score the images.

[0205] It is preferable that the scoring system according to one aspect of the present disclosure further includes a human image detection unit that extracts image features from the received image to detect a person, and that the receiving unit receives the designation information that specifies the image area of ​​the person detected by the human image detection unit.

[0206] According to the above configuration, it is possible to automatically generate designation information for designating an image area of ​​a person based on features, so that it is easy to designate the image area of ​​a person to focus on when outputting an evaluation score for a received image.

[0207] It is preferable that the scoring system according to one aspect of the present disclosure further includes a first application processing unit that performs at least one of classification, sorting, and selection processing on the received image based on predetermined conditions related to the evaluation score output by the evaluation score output unit.

[0208] According to the above configuration, it is possible to easily classify, sort, select, and so on, the received images based on the scoring results.

[0209] It is preferable that the scoring system according to one aspect of the present disclosure further includes a text generation unit that generates text describing the content of the received image by using a text generation model that has been trained to generate text describing the content of the image when focusing on a specific person contained in the training image.

[0210] According to the above configuration, text is generated that explains the content of the received image from the perspective of the person who was focused on when the evaluation score was output, so that the user can easily understand the content of the received image, including the reasons why the evaluation score is high or low.

[0211] It is preferable that the scoring system according to one embodiment of the present disclosure further includes a second application processing unit that performs at least one of classification, sorting, and selection processing on the received image based on predetermined conditions related to at least one of the evaluation score output by the evaluation score output unit and the text generated by the text generation unit.

[0212] According to the above configuration, it is possible to easily classify, sort, select, and so on the received images based not only on the scoring result but also on the text explaining the contents of the received images.

[0213] It is preferable that the scoring system according to one aspect of the present disclosure further includes an attribute information output unit that acquires an image of a person, uses an attribute information output model that has been trained to output attribute information regarding the attributes of the person, and outputs the attribute information for the person that is focused on when the evaluation score is output.

[0214] According to the above configuration, the attribute information of the person who has been focused on is output when the evaluation score is output, so that the user can easily understand the content of the reception image.

[0215] It is preferable that the scoring system according to one embodiment of the present disclosure further includes a third application processing unit that performs at least one of classification, sorting, and selection processing on the received image based on predetermined conditions related to at least one of the evaluation score output by the evaluation score output unit and the attribute information output by the attribute information output unit.

[0216] According to the above configuration, it is possible to easily classify, sort, select, and so on the received images based not only on the scoring result but also on the attribute information of the person who was focused on when the evaluation score was output.

[0217] It is preferable that the scoring system according to one aspect of the present disclosure further includes a first document generation unit that generates a specified document using the received image that has been subjected to at least one of the selection, classification, or sorting processes.

[0218] According to the above configuration, the received images are classified, sorted, selected, etc. based on the scoring results, etc., so that documents (daily reports, albums, etc.) can be automatically generated using appropriate received images, thereby reducing the workload of the user.

[0219] It is preferable that the scoring system according to one aspect of the present disclosure further includes an image cropping unit that crops out an image of a person that has been focused on when the evaluation score is output based on the text generated by the text generation unit and the evaluation score.

[0220] According to the above configuration, the image is cropped based on the content of the text that explains the reason for the high evaluation score, etc., so that the image is automatically trimmed to a more appropriate range, such as an image that is cropped to capture the area that shows the person having fun playing with friends without being too much or too little.

[0221] It is preferable that the scoring system according to an aspect of the present disclosure further includes a second document generating unit that generates a document using the image of the person cut out by the image cutout unit.

[0222] According to the above configuration, a document (daily report, album, etc.) can be automatically generated from an image in which an appropriate range has been trimmed from the received image, thereby reducing the workload of the user.

[0223] It is preferable that the scoring system according to one aspect of the present disclosure further includes a transmission unit that transmits the image of the person cut out by the image cutout unit to an information processing terminal owned by a related person of the person.

[0224] According to the above configuration, an image with an appropriate range cut out is automatically sent to related parties (guardians, etc.) of the person (child) appearing in the image as appropriate, allowing guardians, etc. to properly check how their child is doing at the facility, etc., and therefore feel safe leaving their child at the facility and watching over them.

[0225] It is preferable that the scoring system according to one aspect of the present disclosure further includes a learning unit that trains the image scoring model.

[0226] According to the above configuration, it is possible to generate a trained model used to achieve the above effects.

[0227] It is preferable that the scoring system according to one aspect of the present disclosure further includes an alert unit that presents an alert when the evaluation score output by the evaluation score output unit meets a predetermined condition.

[0228] According to the above configuration, for example, when an alert is presented to the user that the evaluation score is low, the user can retake the photo of the child in question.

[0229] It is preferable that the scoring system according to one aspect of the present disclosure further includes a storage processing unit that performs processing to store in a storage device only those received images for which the evaluation score output by the evaluation score output unit meets predetermined conditions.

[0230] According to the above configuration, for example, only images with high evaluation scores can be stored in a storage (memory device), so that storage capacity of the storage can be saved.

[0231] A method according to one aspect of the present disclosure includes a receiving step of receiving an image containing at least one person as a received image using a scoring system, and an evaluation score output step of acquiring an image containing one or more people as a training image and outputting an evaluation score for the received image using an artificial intelligence model trained to output an evaluation score for the training image when focusing on a specific person included in the training image.

[0232] According to the above configuration, the same effects as the above-mentioned scoring system can be achieved.

[0233] A program according to one aspect of the present disclosure is a program for causing a computer to function as the scoring system described above, and causes the computer to function as each of the above-mentioned parts.

[0234] According to the above configuration, the same effects as the above-mentioned scoring system can be achieved.

[0235] One aspect of the present disclosure is useful in an information processing system for scoring images. [Explanation of symbols]

[0236] 1. Scoring System 10, 10A Server 20, 20A User terminal 11, 21, 11A, 21A storage section 12, 22, 12A, 22A Control unit 23A Imaging unit 101, 201 processors 102, 202 Main memory 103, 203 Auxiliary memory 104, 204 Communication Interface 110, 210 Bus 121, 221 Reception 122, 222 Evaluation score output section 123, 223 Specification information generation unit 124, 224 Person image detection unit 125, 225 Text Generation Section 126, 226 Attribute information output section 127, 227 Image Cutout 128, 228 Transmitter 129, 229 First application processing unit 130, 230 Second application processing unit 131, 231 Third application processing section 132, 232 First document generation unit 133, 233 Second Document Generation Section 134, 234 Learning Department 235 Alert Department 236 Storage Processing Unit 205 Input / Output Interface 206 Input Device 207 Output Device M, M' artificial intelligence model M1, M1' image scoring models M2, M2' text generation models M3, M3' Attribute information output model

Claims

1. a reception unit that receives an image including at least two persons as a received image; an evaluation score output unit that acquires an image including two or more persons as a learning image, and outputs an evaluation score for the received image using an image scoring model that has been trained to output an evaluation score for the learning image when focusing on a specific person included in the learning image; The evaluation score is a score including an evaluation of the communication situation between the specific person and other people. Scoring system.

2. The receiving unit receives designation information that designates an image area of ​​a person included in the received image, the image scoring model has been trained to acquire learning designation information that designates an image region of a person included in the learning image and output an evaluation score for the learning image; The scoring system according to claim 1 , wherein the evaluation score output unit uses the image scoring model to output the evaluation score for the received image based on the specified information.

3. The scoring system according to claim 2 , further comprising a specific information generating unit that generates the specific information for each of a plurality of persons included in the reception image in sequence.

4. A person image detection unit extracts image features from the reception image to detect a person, The scoring system according to claim 2 , wherein the receiving unit receives the designation information that designates an image area of ​​a person detected by the person image detection unit.

5. The scoring system of claim 1, further comprising a first application processing unit that performs at least one of classification, sorting, and selection processing on the received image based on predetermined conditions regarding the evaluation score output by the evaluation score output unit.

6. The scoring system of claim 1, further comprising a text generation unit that generates text explaining the content of the received image using a text generation model that has been trained to generate text explaining the content of the image when focusing on a specific person contained in the learning image.

7. The scoring system of claim 6, further comprising a second application processing unit that performs at least one of classification, sorting, and selection processing on the received image based on predetermined conditions related to at least one of the evaluation score output by the evaluation score output unit and the text generated by the text generation unit.

8. The scoring system of claim 1, further comprising an attribute information output unit that acquires an image of a person, uses an attribute information output model that has been trained to output attribute information regarding the attributes of the person, and outputs the attribute information for the person that is focused on when the evaluation score is output.

9. The scoring system of claim 8, further comprising a third application processing unit that performs at least one of classification, sorting, and selection processing on the received image based on predetermined conditions related to at least one of the evaluation score output by the evaluation score output unit and the attribute information output by the attribute information output unit.

10. A scoring system as described in any one of claims 5, 7, or 9, further comprising a first document generation unit that generates a specified document using the received image that has been subjected to at least one of the selection, classification, or sorting processes.

11. The scoring system of claim 6, further comprising an image cropping unit that crops out an image of a person that has been focused on when the evaluation score is output based on the text generated by the text generation unit and the evaluation score.

12. The scoring system according to claim 11 , further comprising a second document generating unit that generates a document using the image of the person cut out by the image cutout unit.

13. The scoring system according to claim 11 , further comprising a transmission unit that transmits the image of the person cut out by the image cutout unit to an information processing terminal owned by a related person of the person.

14. The scoring system of claim 1 , further comprising a learning unit that trains the image scoring model.

15. The scoring system according to claim 1 , further comprising an alert unit that presents an alert when the evaluation score output by the evaluation score output unit meets a predetermined condition.

16. The scoring system of claim 1 , further comprising a storage processing unit that performs processing to store in a storage device only those accepted images for which the evaluation score output by the evaluation score output unit meets a predetermined condition.

17. Using a scoring system, a receiving step of receiving an image including at least two persons as a received image; an evaluation score output step of acquiring an image including two or more persons as a learning image, and outputting an evaluation score for the received image using an artificial intelligence model trained to output an evaluation score for the learning image when focusing on a specific person included in the learning image; The method, wherein the evaluation score is a score including an evaluation of a communication situation between the specific person and other persons.

18. A program for causing a computer to function as the scoring system according to claim 1, the program causing the computer to function as each of said parts.

Images