Computer systems, data processing methods, and programs

Abstract

This invention provides a computer system, data processing method, and program that shorten the time required for data analysis and processing. [Solution] A computer system having one or more computers for analyzing data, wherein each computer has a memory for storing a program and a processor for executing processing according to the program, and the processor, upon receiving data input, performs a detection process to detect a portion of the data that satisfies conditions specified by the user and generates first data indicating the location of the portion in the data; an extraction process to cut out the portion indicated by the first data from the data and generate second data; and an analysis process to analyze the second data.

Description

Technical Field

[0001] The present disclosure relates to a technology for improving the computational efficiency of data.

Background Art

[0002] Methods are known in which artificial intelligence (AI) such as neural networks are used to analyze multimedia data including a plurality of expression media such as images including moving images or still images, texts, and audios. For example, image classification using Vision Transformer (ViT) is known. On the other hand, Retrieval Augmented Generation (RAG) technology, which improves the accuracy of image classification by using individually accumulated data as external information for generative AI, is known. Currently, RAG mainly targets text data, but it is thought that support for multimedia data will progress going forward.

[0003] However, since multimedia data has a large data volume, the amount of accumulated data is large, and the processing time for analysis tends to be long. Therefore, reducing the amount of accumulated multimedia data and shortening analysis processing time are issues.

[0004] An example of a compression technology for reducing the amount of accumulated multimedia data is disclosed in Patent Document 1. The data compression method disclosed in Patent Document 1 achieves high compression of image data by allocating more bits to regions of an object type, which is the type of object specified by a user, to achieve high image quality, while allowing image quality degradation in regions other than the object type. This achieves a high compression rate and reduces the amount of accumulated data.

[0005] On the other hand, for industrial data, it is not always necessary to reproduce all the information contained in the data with high fidelity after compression and decompression. For example, in the case of inspecting power transmission towers using video data captured by a drone, high image quality is required for the area showing the power transmission towers, but image quality degradation is acceptable for areas such as vegetation in the background. It is conceivable to apply the technology disclosed in Patent Document 1 to this video data to control the bit allocation so that the area containing the object species is made high quality and the area without the object species is highly compressed. In this case, the image quality necessary for inspecting the object species can be obtained, and the amount of data to be stored can be reduced. [Prior art documents] [Patent Documents]

[0006] [Patent Document 1] Japanese Patent Publication No. 2024-145685 [Overview of the project] [Problems that the invention aims to solve]

[0007] When the data compression technology disclosed in Patent Document 1 is applied to image data, the amount of data stored can be reduced, but it cannot be expected to shorten the time required for analysis. This is because the image data being analyzed is uncompressed. This problem occurs not only when the target of analysis is image data, but also when it comes to other data such as text or audio.

[0008] One of the purposes included in this disclosure is to provide a computer system, a data processing method, and a program that reduces the time required for data analysis and processing. [Means for solving the problem]

[0009] A computer system according to one aspect of the present disclosure is a computer system having one or more computers and analyzing data, wherein each computer has a memory for storing a program and a processor for executing processing according to the program, and the processor, upon receiving data as input, performs a detection process for detecting a portion of the data that satisfies conditions specified by a user and generating first data indicating the location of the portion in the data; a clipping process for cutting out the portion indicated by the first data from the data and generating second data; and an analysis process for analyzing the second data. [Effects of the Invention]

[0010] According to one aspect of this disclosure, the analysis process only needs to analyze second data, which consists of the portion of the input data that satisfies conditions specified by the user, thus reducing the time required for the analysis process. [Brief explanation of the drawing]

[0011] [Figure 1] This is a block diagram showing an example configuration of the computer system of Example 1. [Figure 2] This figure shows an example of the hardware configuration of the server in Example 1. [Figure 3] Figure 2 shows an example of the information stored in the memory shown. [Figure 4] Figure 3 shows an example of an object specification information management table. [Figure 5] Figure 1 is a flowchart showing an example of the operation procedure of the detection processing unit. [Figure 6] Figure 1 is a flowchart showing an example of the operation procedure of the cutout processing unit. [Figure 7] This figure shows an example of a detected image represented in 2D coordinates. [Figure 8] This figure shows an example of the processing flow of the clipping processing unit and the analysis processing unit in Modification Example 1. [Figure 9] It is a diagram illustrating an example of a processing flow of a clipping processing unit and an analysis processing unit in Modification 2. [Figure 10] It is a block diagram illustrating one configuration example of a computer system according to Embodiment 2. [Figure 11] It is a block diagram illustrating one configuration example of a computer system according to Embodiment 3. [Figure 12] It is a block diagram illustrating one configuration example of a computer system according to Embodiment 4. [Mode for Carrying Out the Invention]

[0012] Examples of the computer system of the present embodiment will be described with reference to the drawings. In the following description, identical or similar configurations or functions are denoted by identical reference numerals, and overlapping descriptions are omitted. [Examples]

[0013] (Configuration) An overview of the computer system according to Embodiment 1 will be described with reference to FIG. 1. FIG. 1 is a block diagram illustrating one configuration example of the computer system according to Embodiment 1. The computer system 100 includes a data generation source 110, a server 10, and a storage device 120. The server 10 is connected to the storage device 120. An object type designation interface 101 is connected to the server 10.

[0014] The data generation source 110 is an entity that generates data to be compressed. The data generation source 110 is, for example, an image sensor that generates moving images. In Embodiment 1, for the sake of simplicity of explanation, the case where the data generation source 110 is an image sensor that generates moving images will be described, but the data generation source 110 is not limited to an image sensor that generates moving images, and the generated data is not limited to moving images.

[0015] For example, the data generation source 110 may be an image sensor that generates still images, or may be a sensor such as a vibration sensor that generates one-dimensional time-series data. The data generation source 110 is not limited to sensors, and may be configured by a software program and a processor that executes the program. The software program is, for example, a software program such as computer graphics that generates moving images or still images.

[0016] Further, the data generated by the data generation source 110 may be data obtained by processing data generated by a sensor or a software program. The processed data is, for example, a Segmentation Map obtained by applying a Semantic Segmentation machine learning model (hereinafter referred to as an SS model) to each frame of a moving image. The SS model is an example of a model for image classification. Although it is difficult to distinguish overlapping objects, it has the advantage of being able to detect irregular regions such as the sky or roads. The data generated by the data generation source 110 may be a moving image file or the like stored in a recording device. Further, although FIG. 1 shows a case where there is one data generation source 110, a plurality of data generation sources 110 may be provided in the computer system 100.

[0017] The object type designation interface 101 is an interface for a user to designate data representing the type of an object. The object type designation interface 101 is configured by, for example, an input device such as a mouse, keyboard, or touch panel, an output device such as a display or a speaker, and a software program executed by a processor. The data representing the type of an object is, for example, object type designation information 11. Details of the object type designation information 11 will be described later.

[0018] The storage device 120 is a storage device that stores image data such as video data 121 generated by the data source 110. The storage device 120 is either an HDD (Hard Disk Drive) or an SSD (Solid State Drive). The storage device 120 is not limited to a block device composed of an HDD or SSD; it may also be a file storage device, a content storage device, or a volume built on a storage system. The storage device 120 can be any device that stores data in any way.

[0019] The video data 121 may be directly stored in the storage device 120 by the data source 110, or it may be received from the data source 110 by the server 10 and then stored in the storage device 120 by the server 10. The video data 121 may be tensor data in which each frame 17 is represented in Raw, or it may be compressed data according to a video compression standard such as H.264. The video data 121 may also be data composed of any bit sequence that represents the video as an additional data structure. Below, we will explain the case where each of the multiple frames 17 constituting the video data 121 is tensor data represented in Raw. Specifically, we will explain the case where the image data of one frame 17 is 3D tensor data composed of height × width × RGB.

[0020] The configuration of the server 10 shown in Figure 1 will be described. Server 10 is an example of an information processing device that executes the data processing method of Embodiment 1. Server 10 has a detection processing unit 12, a clipping processing unit 13, and an analysis processing unit 14. Server 10 also stores object type specification information 11 as management information. Object type specification information 11 consists of a pair of, for example, a sample image 15 in which an object type specified by the user is depicted, and a mask image 16 which is a mask data image indicating the region in the sample image 15 in which the object type is depicted. Hereafter, this pair will be referred to as a support set.

[0021] The object type specification information 11 shown in Figure 1 is an example where the object type is "power transmission tower". The mask data in Figure 1 shows a mask image 16 that covers the "power transmission tower" portion, but the mask data may also be text data such as "power transmission tower". The object type specification information 11 is set by the user via the object type specification interface 101.

[0022] The detection processing unit 12 detects the region of the object type specified by the object type specification information 11 for each of the multiple frames 17 that make up the video data 121, and sends the detection result 18 to the cropping processing unit 13. For example, the detection processing unit 12 inputs the object type specification information 11 and the frames 17 to a deep learning model of Semantic Segmentation (hereinafter referred to as the FSS model) that utilizes the technique of few-shot learning. The FSS model is a model that improves prediction accuracy by training the SS model with a small number of labeled samples.

[0023] The FSS model converts the image data of frame 17 into binary data consisting of a first value representing a pixel that represents an object type and a second value representing a pixel that represents an area other than the object type, and outputs it. The detection processing unit 12 sends the output of the FSS model as the detection result 18 to the cropping processing unit 13. The detection result 18 is a two-dimensional tensor data of height × width × 2 values ​​for frame 17. Hereinafter, for each pixel in the detection result 18, the first value indicating that an object type has been detected is expressed as positive, and the second value indicating that an object type has not been detected is expressed as negative. The two-dimensional tensor data of the detection result 18 corresponds to the first data.

[0024] Furthermore, if the video data 121 stored in the storage device 120 is compressed data, the frame 17 input to the server 10 is decompressed by a decoder (not shown). A detailed explanation of the decoder's processing is omitted here.

[0025] The process of detecting the region of an object type from frame 17 is not limited to the process using the FSS model. The detection processing unit 12 may, instead of the FSS model, use, for example, an SS model that detects the region of a pre-hardcoded object type in frame 17. Alternatively, the detection processing unit 12 may detect a region of predetermined brightness as the region of an object type in frame 17. In this case, the object type specification interface 101 and object type specification information 11 do not need to be provided. The object type detection process is not limited to the methods described above; any process that takes frame 17 as input and outputs a detection result 18 is acceptable.

[0026] When the cropping processing unit 13 receives the frame 17 and the detection result 18 as input, it crops the image containing the region of positive pixels where an object type was detected in the detection result 18 from the image of frame 17. For example, the cropping processing unit 13 crops the smallest rectangle containing positive pixels in the detection result 18 from the frame 17. The cropping processing unit 13 sends the cropped image 19, which is the image cropped from frame 17, to the analysis processing unit 14. The cropped image 19 corresponds to the second data. Note that the cropping process is not limited to the method described above.

[0027] When the analysis processing unit 14 receives the cropped image 19 from the cropping processing unit 13, it performs analysis on the cropped image 19 and outputs the analysis result 102. For example, the analysis processing unit 14 classifies the image using a neural network such as ResNet (Residual Network) or ViT. Specifically, the analysis processing unit 14 inputs the cropped image 19 into a neural network model and outputs the image classification processing result output from the neural network as the analysis result 102. Alternatively, when the analysis processing unit 14 receives the cropped image 19 from the cropping processing unit 13, it may use a neural network to perform a caption generation process that generates text explaining the content displayed in the cropped image 19. However, the analysis processing unit 14 is not limited to performing the processing in the manner described above, and may perform the analysis processing in any other manner.

[0028] Next, an example of the hardware configuration of server 10 will be described with reference to Figure 2. Figure 2 is a diagram showing an example of the hardware configuration of the server in Embodiment 1. Figure 2 shows the hardware configuration of server 10, which performs the processing of each part described with reference to Figure 1 and holds the management information 25. Server 10 has a processor 21, a switch 22, memory 23, a front-end interface 20, and a back-end interface 26. The processor 21 is connected to the memory 23, the front-end interface 20, and the back-end interface 26, respectively, via the switch 22.

[0029] The front-end interface 20 is an interface for connecting the server 10 to the data generation source 110 and the management terminal 201. The management terminal 201 is an information processing device such as a PC (Personal Computer) operated by a user. The object type specification interface 101 shown in Figure 1 is provided on the management terminal 201. The back-end interface 26 is an interface for connecting the server 10 to the storage device 120.

[0030] The processor 21 serves as a control unit that controls the entire server 10 and as an arithmetic unit that executes the data processing method of Embodiment 1. The processor 21 is a general-purpose arithmetic unit, an accelerator, or a hardware encoder / decoder, or a combination thereof. The general-purpose arithmetic unit is a CPU (Central Processing Unit). The accelerator is a GPU (Graphical Processing Unit) or FPGA (Field Programmable Gate Array), etc. The hardware encoder / decoder is, for example, a hardware encoder / decoder for a standard codec such as HEVC (High Efficiency Video Codec).

[0031] Memory 23 stores the program 24 executed by the processor 21 and the management information 25 used by the processor 21 when executing the program 24. Memory 23 is also used as a work area. Memory 23 is a non-volatile memory such as flash memory. Memory 23 may also store programs such as the OS (Operating System) and information related to the program.

[0032] Figure 3 shows an example of the information stored in the memory shown in Figure 2. Memory 23 stores the detection program 24, the clipping program 241, the clipping program 242, and the analysis program 243. The detection program 241 is a program that implements the detection processing by the detection processing unit 12. The clipping program 242 is a program that implements the clipping processing by the clipping processing unit 13. The analysis program 243 is a program that implements the analysis processing by the analysis processing unit 14. Memory 23 stores the object specification information management table 301 as management information 25. The object specification information management table 301 is data for managing the object type specification information 11.

[0033] The processor 21 executes the detection program 241, thereby executing the functions of the detection processing unit 12 shown in Figure 1. The processor 21 executes the clipping program 242, thereby executing the functions of the clipping processing unit 13 shown in Figure 1. The processor 21 executes the analysis program 243, thereby executing the functions of the analysis processing unit 14 shown in Figure 1. A configuration including at least the processor 21 and memory 23 functions as a single computer.

[0034] Figure 3 shows the case where program 24 is stored in memory 23, but program 24 may be pre-stored on a recording medium (not shown), such as a DVD (Digital Versatile Disc), and installed from the recording medium to server 10. Alternatively, program 24 may be stored on the HDD or SSD of an external server (not shown) and distributed from the external server to server 10 via a network such as the Internet. Furthermore, the neural network, such as the FSS model, may be stored in memory 23 as shown in Figure 2, or it may be stored on an external server (not shown). If the neural network is stored on an external server (not shown), the processor 21 can use the neural network by connecting to the external server via a network such as the Internet.

[0035] Figure 4 shows an example of the object specification information management table shown in Figure 3. The object specification information management table 301 is, for example, a table of data. The object specification information management table 301 has fields for data source ID 401, object type ID 402, and object type specification information 403. The object type specification information 403 has fields for sample image 404 and mask image 405. The fields included in the entries shown in Figure 4 are examples and are not limited to the items shown in Figure 4.

[0036] The data source ID 401 is a field that stores the identifier of the data source 110. The identifier of the data source 110 is, for example, a string set by the user, or a MAC (Media Access Control) address or IP (Internet Protocol) address assigned to the data source 110. The identifier of the data source 110 may also be an address on the storage device 120, a path, or any code that can identify the data source 110. However, if there is only one data source 110, the user does not need to specify one data source 110 from among multiple data sources 110, so the entry does not need to have a field for data source ID 401.

[0037] Object type ID 402 is a field that stores the identifier of the object type. The object type identifier is, for example, a string set by the user. The object type identifier is not limited to being set by the user. If there is only one object type specification information 403 set for a single type of object, it is not necessary to provide an object type ID 402.

[0038] The object type specification information 403 is a field that stores two images for specifying the object type, corresponding to the object type ID 402 of each data source ID 401. The two images are a sample image 404 and a mask image 405. For example, the sample image 404 is an image containing the object type specified by the user. The mask image 405 is a monochrome image in which a predetermined value is assigned to the area in the sample image 404 that contains the object type specified by the user.

[0039] Note that the mask image 405 is not limited to the configuration described above. The mask image 405 may be XML (Extensible Markup Language) containing the coordinates and size of a bounding box representing the location of the area where the object type is depicted. Furthermore, the mask image 405 is not limited to the XML data format, but may be any other arbitrary data format. The object type specification information 403 may be text data representing the object type in natural language. Furthermore, the object type specification information 403 is not limited to the support set consisting of the sample image 404 and the mask image 405, but may also be data obtained by converting the above text data.

[0040] Here, with reference to Figure 4, a specific example of the information stored in the object specification information management table 301 will be explained. Figure 4 shows a case where, for a data source to which the identifier "110A" is assigned, two types of object types, "obj1" and "obj2," are specified by the object type specification information registered in rows 406 to 408.

[0041] The object type assigned to identifier "obj1" is a power transmission tower. Power transmission towers are specified by two sets of object type specification information: the support set of sample image 410 and mask image 420, and the support set of sample image 411 and mask image 421. For example, sample image 410 contains both power transmission towers and wind turbines, but mask image 420 specifies that it is a power transmission tower. On the other hand, the object type assigned to identifier "obj2" is a wind turbine. Wind turbines are specified by object type specification information from the support set of sample image 412 and mask image 422.

[0042] Figure 4 shows a case where the data source ID 401, object type ID 402, and object type specification information 403 are represented in a single table, but the relationships between these items are not limited to being represented in a single table. The relationships between these items may be represented by dividing them into two tables: a first table that associates the data source ID 401 and object type ID 402, and a second table that associates the object type ID 402 and object type specification information 403. In this case, the object type specification information 403 for any object type can be shared by multiple data sources 110.

[0043] Furthermore, the object specification information management table 301, as explained with reference to Figure 4, is just one example; any data structure capable of managing information specifying object types is acceptable. The data structure for managing information specifying object types may be a data structure other than a table, such as XML, YAML (YAML Ain't Markup Language), a hash table, or a tree structure.

[0044] Furthermore, the hardware configuration of server 10, as described with reference to Figure 2, may be a configuration combining multiple electronic components or multiple semiconductor devices, or it may be composed of a single IC (Integrated Circuit). All or part of the detection processing unit 12, the cutting processing unit 13, and the analysis processing unit 14 shown in Figure 1 may be composed of an ASIC (Application Specific Integrated Circuit) or an FPGA. Server 10 may be implemented as a VM (Virtual Machine) whose functions are performed by virtualization technology. Also, server 10 may be implemented using container technology, which divides the server's processing unit into multiple partitions and allows multiple application software programs to run simultaneously. Moreover, server 10 may have other hardware configurations, not limited to the configuration shown in Figure 2.

[0045] Furthermore, in Example 1, the data source 110, management terminal 201, server 10, and storage device 120 are separate hardware configurations. The computer system 100 may be an information processing device in which these configurations are integrated. In this case, each configuration may be implemented by a VM running on the same computer, by different containers running on the same OS, or by an application software program running on the same OS. The multiple implementation forms that constitute the computer system 100 are not limited to this implementation form, although in Example 1 the data source 110 is an image sensor, the server 10 is an edge device, and the storage device 120 is a device with an HDD.

[0046] Furthermore, in Embodiment 1, as described above, the processor 21 executes a program stored in the memory 23, thereby executing the functions of the detection processing unit 12, the clipping processing unit 13, and the analysis processing unit 14. The computer system 100 may have multiple computers (each computer including a processor 21 and memory 23), and the functions of each part may be executed by separate computers.

[0047] (operation) Next, the operation of the computer system 100 of Example 1 will be described. First, the operation of the detection processing unit 12 shown in Figure 1 will be described with reference to Figure 5. Figure 5 is a flowchart showing an example of the operation procedure of the detection processing unit shown in Figure 1.

[0048] The detection program 241 is a program that takes frame 17 as input and outputs a detection result 18. The processor 21 executes the detection program 241, for example, when it reads frame 17 from the storage device 120. As a result, the processor 21 operates according to the procedure shown in the flowchart of Figure 5.

[0049] In step S501, the processor 21 extracts rows from the object-specific information management table 301 that correspond to the data source ID of the frame 17. For example, consider the case where the object-specific information management table 301 contains the information shown in Figure 4, and the identifier of the data source 110 of frame 17 is "110A". In this case, in step S501, the processor 21 extracts rows 406 to 408 from the object-specific information management table 301.

[0050] Steps S502 to S505 are loop processes that repeat the same processing for the object type IDs included in the rows extracted in step S501. For example, for the table shown in Figure 4, the processing in steps S503 to S504 is executed for both the case where the object type ID is "obj1" and the case where the object type ID is "obj2". In other words, the processor 21 performs the loop process twice.

[0051] In step S503, the processor 21 obtains a set of sample image 404 and mask image 405 pairs corresponding to the object type ID of interest in the current iteration from the object specification information management table 301. For example, consider the case where the object type ID = "obj1" in the object specification information management table 301 shown in Figure 4. In this case, in step S503, the processor 21 obtains two support sets: a support set consisting of sample image 410 and mask image 420, and a support set consisting of sample image 411 and mask image 421.

[0052] In step S504, the processor 21 uses the set of sample images 404 and mask images 405 acquired in step S503 as a support set, and the frame 17 as a query image, inputting the support set and the query image into the deep learning model. The deep learning model is a model that classifies images from the query image based on the support set, and is, for example, an FSS model.

[0053] The deep learning model takes a query image and a support set as input and outputs a 2D tensor data for each pixel of the query image, indicating whether or not it contains an object type specified by the support set. In this case, the 2D tensor data is height × width × binary. The binary is, for example, black for the object type and white for the non-object type. The support set may consist of one set of sample images and mask images, or multiple sets. The more support sets input to the deep learning model, the better the detection accuracy of the image classification.

[0054] Once all iterations of the loop processing in steps S502 to S505 have been completed, the processor 21 proceeds to step S506. In step S506, the processor 21 calculates the logical OR of the calculation results from step S504 for each object type ID, pixel by pixel. In other words, the processor 21 generates two-dimensional tensor data in frame 17 that indicates the region where one or more object types specified in the object designation information management table 301 are captured. After obtaining the two-dimensional tensor data generated in step S506 as the detection result 18, the processor 21 terminates the detection program 241.

[0055] Note that the trigger for starting the detection program 241 is not limited to the above case. For example, consider the case where the video data 121 stored in the storage device 120 is video data compressed using a video compression standard such as H.264. In this case, the trigger for starting the detection program 241 is the timing when the processor 21 reads the video data from the storage device 120 and decompresses it into frames 17. The trigger for starting the detection program 241 may be any other arbitrary timing. Also, the deep learning model used in step S504 is not limited to the FSS model. The deep learning model may be, for example, an object detection deep learning model that utilizes the technique of few-shot learning.

[0056] Next, the operation of the cropping processing unit 13 shown in Figure 1 will be explained with reference to Figure 6. Figure 6 is a flowchart showing an example of the operation procedure of the cropping processing unit shown in Figure 1. The cropping program 242 is a program that takes the frame 17 and the detection result 18 as input and outputs a cropped image 19. The processor 21 executes the cropping program 242, for example, when the detection program 241 is completed. As a result, the processor 21 operates according to the procedure shown in the flowchart of Figure 6. Note that the trigger for starting the execution of the cropping program 242 is not limited to the timing of the completion of the detection program 241.

[0057] Furthermore, a two-dimensional coordinate system is assigned to multiple pixels that make up the image data of frame 17 and detection result 18, with the width direction (left-right direction) as the X-axis and the height direction (up-down direction) as the Y-axis. Figure 7 shows an example of a detection result image represented in two-dimensional coordinates. Detection result 18 shows a power transmission tower.

[0058] In step S601, the processor 21 obtains the coordinates (x1, y1) of the leftmost pixel and the coordinates (x2, y2) of the rightmost pixel among the positive pixels in the 2D coordinate system to which the detection result 18 is applied. In step S602, the processor 21 obtains the coordinates (x3, y3) of the topmost pixel and the coordinates (x4, y4) of the bottommost pixel among the positive pixels in the 2D coordinate system. For illustrative purposes, Figure 7 shows the top, bottom, left, and rightmost pixels of the positive pixels as black circles.

[0059] In step S603, the processor 21 crops a rectangular region from the original frame 17, with coordinates (x1, y3) as the top left and coordinates (x2, y4) as the bottom right, to obtain a cropped image 19. The rectangular cropped image 19 has two vertically parallel sides that are parallel to the two vertically parallel sides of the frame 17, and two horizontally parallel sides that are parallel to the two horizontally parallel sides of the frame 17. The cropped image 19 is a region where the minimum coordinate value of a positive pixel is the lower limit and the maximum coordinate value of a positive pixel is the upper limit for each height and width dimension of the 2D tensor data. After obtaining the cropped image 19 in step S603, the processor 21 terminates the cropping program 242.

[0060] As a result, all pixels determined to be positive in the detection process described with reference to Figure 5 are included in the cropped image 19. In addition, the smallest rectangle on the outer perimeter of frame 17, where each of the four outer edges is parallel to the corresponding four edges, can be cropped from frame 17 as the cropped image 19.

[0061] Furthermore, the method for cropping the region representing the pixels determined to be positive from frame 17 is not limited to the method described with reference to Figures 6 and 7. For example, the smallest rectangle containing the pixels that were found to be positive in the detection result 18 may be tilted at an angle in frame 17. In other words, the cropped image 19 does not need to have each of its four outer edges parallel to each of the four outer edges of frame 17.

[0062] After the cropping process, the processor 21 executes the analysis program 243, for example, triggered by the completion of the cropping program 242. The processor 21 performs analysis on the cropped image 19 and outputs the analysis result 102. For example, the processor 21 classifies the image using a neural network. A detailed explanation of the analysis process is omitted here. The processor 21 may send the analysis result 102 to the management terminal 201 via the front-end interface 20, or it may store it in the storage device 120. After that, the processor 21 completes the analysis program 243.

[0063] According to the computer system 100 of Embodiment 1, the analysis processing unit 14 analyzes the cropped image 19, which has a smaller data volume than the original frame 17, thus reducing the analysis processing time compared to analyzing the original frame 17. Furthermore, Embodiment 1 has the advantage that unnecessary areas are removed from the cropped image 19 by allowing the user to specify the necessary object types for the analysis processing unit 14 via the object type specification interface 101. For example, consider a case where the analysis processing unit 14 receives an image of a power transmission tower as input and outputs an inspection result 102 as a binary value of pass or fail for the power transmission tower. When object type specification information 11 representing a power transmission tower is specified by the user, areas unrelated to the power transmission tower are removed from the frame 17, leaving the power transmission tower area in the cropped image 19. The cropped image 19 retains the power transmission tower area necessary for the inspection of the power transmission tower, while other areas that are unnecessary for the inspection and could become noise for the analysis processing unit 14 are removed. Therefore, it is believed that the analysis processing unit 14 can perform analysis with the same or better accuracy as when the original frame 17 is input, simply by receiving the cropped image 19. Thus, the analysis processing time can be shortened while maintaining the accuracy of the analysis processing.

[0064] Furthermore, conventionally, in analysis processing, images are sometimes resized to a fixed resolution before processing. In this case, even if the image input to the analysis processing unit 14 is a cropped image 19, the resolution of the image input to the neural network, etc., does not decrease, so the processing time appears to remain unchanged. However, in Example 1, areas unrelated to the analysis processing are cropped from frame 17, and information that could become noise unnecessary for analysis is removed, so the analysis accuracy is expected to improve. Here, we assume that the higher the resized resolution in the analysis processing unit 14, the higher the analysis accuracy. In this case, if the cropped image 19 is used as input to the analysis processing unit 14, it is expected that the same accuracy as when the original frame 17 is used as input can be achieved even if the image is resized to a smaller resolution. Therefore, even in analysis processing where the image is resized to a fixed resolution, the processing time can be shortened by Example 1 under the condition that the analysis accuracy is the same.

[0065] In Example 1, we described a case where an object type specified by the user is extracted from image data, but the extracted portion is not limited to images of object types. The computer system 100 may also perform processing to extract information necessary for the user from a vast amount of text data. For example, the computer system 100 may be applied to obtain information about product performance from product specifications while shortening the analysis processing time.

[0066] (Variations in cropping) (Variation 1) Modification 1 involves the analysis processing unit 14 dividing the cropped image 19 into multiple patches, converting each patch into a token, and then performing the analysis. Figure 8 shows an example of the processing flow of the cropping processing unit and analysis processing unit in Modification 1. The analysis processing unit 14a in Modification 1 includes a patch division processing unit 71 and a transformer 72. The transformer 72 is an example of a neural network.

[0067] As explained with reference to Figure 6, the cropping processing unit 13 converts the frame 17 and detection result 18 into a cropped image 19. When the patch division processing unit 71 receives the cropped image 19 from the cropping processing unit 13, it converts the cropped image 19 into a set of tokens 701, which is a set of data corresponding to a patch of a predetermined size, using an image classification technique such as ViT. For example, the patch division processing unit 71 divides the cropped image 19 into square patches of 14 pixels vertically x 14 pixels horizontally in a tile-like manner. Next, the patch division processing unit 71 converts each patch into a vector of a predetermined length by a convolution operation or the like. Then, the patch division processing unit 71 converts the vector into a set of tokens 701 in which positional information is embedded in each token by positional encoding. When the transformer 72 receives the set of tokens 701 from the patch division processing unit 71, it performs analysis processing on each token as one word and outputs the analysis result 102.

[0068] According to Modification 1, if the specified object type is, for example, a car, the cropping processing unit 13 deletes the area where the car is not visible. Therefore, the analysis processing unit 14a can reduce the analysis processing time compared to when the entire frame 17 is the target of the analysis processing.

[0069] (Modification 2) Modification 2 is an improvement on Modification 1. Figure 9 shows an example of the processing flow of the cutout processing unit and analysis processing unit of Modification 2. The analysis processing unit 14a of Modification 2 has a patch division processing unit 71 that functions as a preceding processing unit and a transformer 72 that functions as a subsequent processing unit. The cutout processing unit 13 is provided between the patch division processing unit 71 and the transformer 72.

[0070] As shown in frame 17 in Figure 8, there are cases where two cars are visible separately in frame 17. In this case, areas where cars are not visible, such as the area between the two cars, may remain in the cropped image 19.

[0071] Therefore, variations of the clipping processing unit 13, as shown in Figure 9, can be considered. First, the patch division processing unit 71 converts the entire frame 17 into a set of tokens 801. Next, the clipping processing unit 13 selects only the tokens 802 corresponding to patches in which one or more positive pixels exist from the detection results 18. In the example shown in Figure 9, there are multiple tokens 802. The clipping processing unit 13 sends the set of selected tokens 802 to the transformer 72. The transformer 72 performs analysis processing in the same manner as in Modification 1 and outputs the analysis results 102.

[0072] According to Modification 2, for example, only tokens 802 corresponding to patches showing automobiles are processed by the transformer 72, thus improving computational efficiency compared to the case described with reference to Figure 8. In Modification 2 as well, since the patch division processing unit 71 embeds location information into each token of the token set 801, the selected tokens 802 also retain location information. Therefore, even if the positive pixels in the detection result 18 are discontinuously distributed, for example, if the detection result 18 represents two automobiles located at different locations, the selected tokens 802 retain the location information of the two automobiles in the frame 17. As a result, the transformer 72 can perform an analysis that takes into account the location information of the automobiles in the entire frame 17.

[0073] Furthermore, the cropping processing unit 13 may select some tokens from other patches in addition to the token 802 corresponding to the patch in which the car is visible. For example, the tokens to be added can be selected by adding tokens from a fixed region that the transformer 72 focuses on, regardless of the image, to the cropping processing unit 13. For example, when one or more images are input to the analysis processing unit 14a, the cropping processing unit 13 obtains the Attention value at the position of each layer or token of the transformer 72. Next, the cropping processing unit 13 obtains the average of these Attention values ​​for the position of each token. Token positions where these average values ​​are greater than a threshold can be said to be regions that the transformer 72 focuses on regardless of the image. Therefore, the cropping processing unit 13 selects the token at that position from the token set 801 and adds it to the token 802 for the car region. The threshold is, for example, 1 / [number of tokens constituting the token set 801]. This allows the transformer 72 to identify the location of the token to focus on with a higher probability than randomly focusing on each token. The cropping processing unit 13 may also select the locations of tokens where the Attention value in some layers of the transformer 72 exceeds a threshold. Note that the method described here is just one example; for example, a specific number of additional tokens could be randomly selected from the token set 801, excluding those corresponding to patches showing cars. Combinations of these methods are also possible.

[0074] Note that the analysis processing unit 14a, which is composed of the patch division processing unit 71 and the transformer 72, is an example of the configuration in modified example 2. Any analysis process that divides image data into a set of subsets and processes the set of subsets to obtain the analysis result 102 can be performed in the same way as the process described with reference to Figure 9.

[0075] The computer system 100 of Example 1 has one or more computers. Each computer has a memory 23 for storing a program and a processor 21 for executing processing according to the program. The processor 21 performs detection processing, clipping processing, and analysis processing by executing the program. The detection processing is the process of detecting a portion of the data that satisfies conditions specified by the user when data is input, and generating first data indicating the location of the above portion in the data. The clipping processing is the process of clipping the above portion indicated by the first data from the data and generating second data. The analysis processing is the process of analyzing the second data.

[0076] According to Example 1, in the analysis process, it is only necessary to analyze the cropped image 19, which is a second data set obtained by cropping out the portion of the input image data that satisfies the conditions specified by the user, thus shortening the time required for the analysis process. [Examples]

[0077] Example 1 is a case in which the analysis processing unit 14 performs analysis on the video data 121 and outputs the analysis result 102. Example 2 is a case in which a database that functions as external information is constructed in RAG using the video data 121. In Example 2, the same reference numerals are used for the same components as in Example 1, and their detailed explanations are omitted.

[0078] The configuration of the computer system in Example 2 will be described with reference to Figure 10. Figure 10 is a block diagram showing an example configuration of the computer system in Example 2. The computer system 100a in Example 2 has a server 10a instead of the server 10 shown in Figure 1. Server 10a has a detection processing unit 12, a clipping processing unit 13, a feature generation unit 91, and a vector database construction unit 92. In Example 2, the feature generation unit 91 functions as an analysis processing unit.

[0079] When image data is input to the feature generation unit 91, it outputs embeddings, which are vectors representing the features of the region of an object type. For example, the feature generation unit 91 processes the image data using ViT, which is built on a transformer, and outputs class tokens, which are vectors representing the features of the image. The processing by the feature generation unit 91 is not limited to the above case. In Embodiment 2, the cropped image 19 is input to the feature generation unit 91 as the image data to be analyzed. When the feature generation unit 91 receives the cropped image 19 from the cropping processing unit 13, it calculates embeddings 901 containing the features of the region of an object type specified by the user and outputs them to the vector database construction unit 92.

[0080] The vector database construction unit 92 is implemented by the processor 21 executing a vector database software program. The memory 23 shown in Figure 2 stores the vector database software program. The embedding 901 is input from the feature generation unit 91 to the vector database construction unit 92. The vector database construction unit 92 uses the embedding 901 as the key 911 and the frame identifier 17 as the value 912, and writes the key 911 to the vector database 903, associating the key 911 with the value 912. The frame identifier 17 is, for example, composed of a pair of the file system path of the video data 121 and the serial number of the frame 17 within the video data 121. The frame identifier 17 is not limited to this configuration.

[0081] Alternatively, the vector database construction unit 92 may obtain one embedding 901 for multiple frames 17. In this case, the vector database construction unit 92 may associate the identifier of the video composed of multiple frames 17 as the key 911 of the one embedding 901 with the value 912 and write them to the vector database 903.

[0082] The search process using the vector database 903 returns a value 912 associated with the most similar key 911, for example, based on an indicator such as cosine similarity, for a given vector as a search condition. When a user inputs a search condition to the server 10a, the server 10a reads frames that match the search condition from the vector database 903 and inputs the read frames and search conditions to an external generating AI. The generating AI is, for example, an LLM (Large Language Model). In this case, the user can obtain a more accurate answer from the generating AI. The generating AI may be stored in the storage device 120.

[0083] Modification 2 may also be applied to Example 2. As described with reference to Figure 9, the server 10a may perform a splitting process by the patch splitting processing unit 71, then a cutout process by the cutout processing unit 13, and then perform processing by the transformer 72 to obtain the embedding 901.

[0084] According to Example 2, the processing time can be reduced because features are extracted from the image using ViT during the analysis process. Furthermore, the vector database 903 functions as a search system that returns video frames similar to the search criteria, for example, image, video, or text embedding. In addition, the vector database 903 can be used as external information in RAG. For example, when server 10a receives a question text from the user, it uses the vector database 903 to search for frames that approximate the content of the text. Server 10a then inputs the frames obtained by the search and the user's question text into LLM and provides the LLM output to the user. By performing search extension generation, the accuracy of the LLM's answers can be improved. [Examples]

[0085] Example 3 compresses image data stored in the storage device 120. In Example 3, the same reference numerals are used for the same components as in Examples 1 and 2, and their detailed descriptions are omitted. Furthermore, while Example 3 is described based on the computer system 100a described in Example 2, the base configuration is not limited to that of Example 2.

[0086] (composition) The configuration of the computer system in Example 3 will be described with reference to Figure 11. Figure 11 is a block diagram showing an example configuration of the computer system in Example 3. The computer system 100b in Example 3 has a server 10b instead of the server 10 shown in Figure 1. Server 10b has a compression unit 31 and a RAG construction unit 32.

[0087] The compression unit 31 compresses the video data generated by the data source 110 using video compression technology and stores the data 520 containing the compressed video data in the storage device 120. The video compression technology is, for example, the technology disclosed in Patent Document 1. The compression unit 31 compresses the video data generated by the data source 110. In Embodiment 3, the compression unit 31 compresses the video data generated by the data source 110 by increasing the number of bits allocated to the area of ​​the object type specified by the user and decreasing the number of bits allocated to the area of ​​non-object types. In other words, the compression unit 31 makes the area of ​​the object type high quality and the area of ​​non-object types low quality, thereby reducing the amount of data in each of the multiple frames 17 that make up the video data. This achieves a high compression ratio and reduces the amount of data stored in the storage device 120. The compression unit 31 has a detection processing unit 12 and an encoder 41 that performs compression processing to compress image data.

[0088] The RAG construction unit 32 constructs a vector database 903 using the data 520 stored in the storage device 120. The RAG construction unit 32 includes a decoder 42 that performs decompression processing to decompress compressed image data, a clipping processing unit 13, a feature generation unit 91, and a vector database construction unit 92.

[0089] (operation) When the compression unit 31 receives video data from the data generation source 110, it inputs each frame 17 to the detection processing unit 12. Similar to Embodiment 1, the detection processing unit 12 generates binary data indicating positive regions where an object type specified by the user is depicted and negative regions other than the object type, and outputs the binary data as a detection result 18 to the encoder 41. When the encoder 41 receives the frame 17 and the detection result 18 as input, it enhances the image quality of the portion of the frame 17 corresponding to the positive region of the detection result 18, and reduces the image quality of the portion of the frame 17 corresponding to the negative region of the detection result 18, while also reducing the amount of data in the frame 17. Specifically, the encoder 41 increases the number of bits allocated to the portion of the frame 17 corresponding to the positive region of the detection result 18 and decreases the number of bits allocated to the portion of the frame 17 corresponding to the negative region of the detection result 18. The encoder 41 compresses the frame 17 by controlling the image quality and data amount for each region of the frame 17.

[0090] Subsequently, the encoder 41 stores data 520 in the storage device 120, which associates the compressed video data 521 with the detection results 522 calculated during the compression process. The detection results 522 may be tensor data representing the detection results 18 in raw format, a bit sequence encoded by the Run-length method or the like, or any other data structure. The compressed video data 521 and the detection results 522 in each data 520 may be associated using any data structure. For example, the compressed video data 521 and the detection results 522 may be given the same file name, and these data may be stored in different folders in the storage device 120. Alternatively, for example, the detection results 522 may be embedded as metadata within a video data container defined by a standard such as MPEG-7.

[0091] When the RAG construction unit 32 receives data 520 from the storage device 120, the decoder 42 decodes the data 520 into multiple frames 601 and detection results 602 that constitute the video data. When the clipping processing unit 13 receives the frames 601 and detection results 602 as input, it generates a clipped image 19, for example, as described with reference to Figure 6. Subsequently, the feature generation unit 91 obtains embeddings 901 containing the features of the object type region from the clipped image 19 in the same manner as in Embodiment 2, and outputs the embeddings 901 to the vector database construction unit 92. The vector database construction unit 92 constructs the vector database 903 in the storage device 120 in the same manner as in Embodiment 2.

[0092] According to Example 3, the image data to be analyzed is stored in the storage device in a high-resolution state for the object type region, thus improving the accuracy of the analysis process. Furthermore, the detection result 602 used in the cropping processing unit 13 is the decoded result 18 used by the compression unit 31. Therefore, in the RAG construction unit 32, the detection processing unit 12 does not need to perform processing, and the processing time for RAG construction can be shortened compared to Example 2. Moreover, by using the video compression technology disclosed in Patent Document 1, the video data is compressed and stored in the storage device 120, thus reducing the amount of data stored in the storage device 120 compared to Example 2. As a result, the storage device 120 can be utilized more effectively. [Examples]

[0093] Example 4 implements the server functions described in Examples 1-3 using a group of microservices. In Example 4, the same reference numerals are used for the same components as in Examples 1-3, and their detailed descriptions are omitted. Furthermore, while Example 4 is described based on the computer system 100b described in Example 3, the base configuration is not limited to that of Example 3.

[0094] The configuration of the computer system in Example 4 will be described with reference to Figure 12. Figure 12 is a block diagram showing one example configuration of the computer system in Example 4. The computer system 100c in Example 4 has a server 10c instead of the server 10 shown in Figure 1. Server 10c has a video compression microservice 51, an analysis processing microservice 52, a feature generation microservice 53, and a vector database microservice 54.

[0095] The computer system 100c has a configuration with multiple computers. Each computer has a processor 21 and memory 23 as shown in Figure 2, and executes the functions of one microservice. Figure 12 shows a case where multiple computers are located on a single server 10c. The multiple computers do not have to be located on a single server 10c; they may be connected via a network such as the Internet.

[0096] The video compression microservice 51 is a microservice that performs processing on the compression unit 31 of Embodiment 3. The video compression microservice 51 receives video data from the data source 110, compresses the received video data, and stores data 520 in the storage device 120 that associates the compressed video data 521 with the detection results 522 for each frame.

[0097] The analysis processing microservice 52 is a microservice that executes the processing of the analysis processing unit 14. For example, the analysis processing microservice 52 takes data 520 stored in the storage device 120 as input, converts each frame into a cropped image 19 by cropping processing, performs analysis processing on the cropped image 19, and outputs the analysis result 102.

[0098] The feature generation microservice 53 is a microservice that, upon receiving data 520, performs processing from the decoder 42 to the feature generation unit 91 of the RAG construction unit 32 in Example 3, and outputs embedding 901. The feature generation microservice 53 may also output the embedding 901 to the analysis processing microservice 52, because the processing by the feature generation microservice 53 corresponds to the analysis processing.

[0099] The vector database microservice 54 is a microservice that constructs the vector database 903. The vector database microservice 54 may also function as external information for RAG.

[0100] Note that the microservices shown in Figure 12 are just one example; instead of microservices, virtual machines, software running in user space, libraries running on the same software, or functions on the same software may also be used.

[0101] According to Example 4, multiple microservices can be connected via a network such as the Internet. Therefore, users do not need to provide multiple microservices on a server they manage themselves. For example, a user can select the microservices that provide the necessary functions for their self-managed server from multiple externally provided microservices. As a result, the versatility of the contents disclosed in Examples 1 to 3 is improved.

[0102] The embodiments described above are illustrative for explaining the present invention and are not intended to limit the scope of the present invention to those embodiments described above. Those skilled in the art can implement the present invention in various other forms, such as by changing the specific configuration, without departing from the spirit and intent of the present invention.

[0103] Furthermore, the embodiments described above include the following items. However, the items included in these embodiments are not limited to those listed below.

[0104] (Item 1) A computing system having one or more computers and analyzing data, The aforementioned computer is Memory for storing programs, A processor that performs processing according to the aforementioned program, The processor executes the program, When the aforementioned data is input, a detection process is performed to detect a portion of the data that satisfies the conditions specified by the user and to generate first data indicating the position of the portion in the data. A cropping process that generates second data by cropping out the portion shown by the first data from the aforementioned data, The analysis process described above for analyzing the second data is performed. Computer system.

[0105] According to this approach, in the analysis process, it is only necessary to analyze a second set of data, which consists of the portion of the input data that satisfies the conditions specified by the user, thus shortening the time required for the analysis process.

[0106] (Item 2) In the computer system described in item 1, The aforementioned processor, If the data is video or still image data, the detection process converts the image data into first data which is binary data consisting of a first value indicating a pixel representing an object type specified by the user and a second value indicating a pixel representing an area other than the object type. As part of the cropping process, the second data, which includes a region indicating the first value, is cropped from the image data by referring to the first data. Computer system.

[0107] According to this method, pixels representing the object type specified by the user are recognized in the image data, and the portion of the image data containing the object type is extracted for analysis. Therefore, in the analysis process, only the portion of the image data containing the object type needs to be analyzed, thus reducing the time required for analysis.

[0108] (Item 3) In the computer system described in item 2, The aforementioned processor, If the image data is tensor data, the cropping process involves cropping a region from the image data for each dimension of the height and width of the tensor data, where the lower limit is the smallest coordinate value of the pixel with the first value, and the upper limit is the largest coordinate value of the pixel with the first value, to generate the second data. Computer system.

[0109] According to this, in the image data, the upper and lower edges of the second data region are determined by the coordinates of the pixels with the maximum and minimum values ​​in the height direction, and the left and right edges of the second data region are determined by the coordinates of the pixels with the minimum and maximum values ​​in the width direction.

[0110] (Item 4) In the computer system described in item 2 or 3, The aforementioned processor, If the image data is tensor data, as a preliminary step to the analysis process, the image data is divided into a set of tokens, As part of the aforementioned cropping process, tokens corresponding to patches containing one or more pixels of the first value are cropped from the set of tokens. As a subsequent step in the analysis process, the extracted set of tokens is input into a neural network to obtain the second data from the neural network. Computer system.

[0111] According to this method, even in images containing multiple object types at distant locations, analysis can be performed using data with areas where no object types are present removed. Therefore, the analysis time can be reduced.

[0112] (Item 5) In the computer system described in item 4, The aforementioned processor, In addition to the tokens corresponding to the patch containing one or more pixels of the first value from the set of tokens, The aforementioned neural network extracts tokens at the location it is focusing on. Computer system. This allows the neural network to pinpoint the location of the token to focus on with a higher probability than if it were to randomly focus on each token.

[0113] (Item 6) In a computer system described in any one of items 2 to 5, The storage device has a storage device for storing the aforementioned image data, The aforementioned processor, As part of the analysis process, embeddings including the characteristics of the object type are obtained from the second data. A vector database is constructed in the storage device, with the aforementioned embedding as the key and the identifier of the image data as the value, associating the key with the value. Computer system.

[0114] This creates a vector database on the storage device that stores identifiers for embedded images containing object types specified by the user, associating them with the identifiers of the image data. The vector database can then be used as external information for search, extension, and generation technologies.

[0115] (Item 7) In the computer system described in item 6, The aforementioned processor, Based on the first data which is the detection result of the detection process, a compression process is performed to compress the image data by increasing the image quality of the region of the object type and decreasing the image quality of the region other than the object type, and then the compressed image data is associated with the detection result and stored in the storage device. Using the detection results, a decompression process is performed to decompress the compressed image data. The cropping process is performed using the expanded image data and the detection results. The analysis process is performed on the second data. Computer system.

[0116] This ensures that the image data to be analyzed is stored in the storage device with high-quality resolution for each object type, while also reducing the amount of data accumulated in the storage device. As a result, not only is the accuracy of the analysis process improved, but the storage device can be utilized more effectively.

[0117] (Item 8) In a computer system described in any one of items 2 to 7, The memory stores management information including a support set which is a set of sample images containing the object type to be detected by a deep learning model that classifies images, and mask data indicating the region in the sample image in which the object type is depicted. The aforementioned processor, As part of the detection process, the management information and the image data are input to the deep learning model, and the first data is obtained from the deep learning model. Computer system.

[0118] As a result, the deep learning model receives image data that serves as the query image and management information that includes information about the object type, allowing it to classify the parts of the image data that are closer to the object type.

[0119] (Item 9) In the computer system described in item 8, Having an interface for setting the aforementioned management information, Computer system. This allows users to specify object types using an interface for configuring management information.

[0120] (Item 10) The computer described in any one of items 1 to 9, A storage device for storing the aforementioned data, A computer system having [a certain feature]. This means that the system has a storage device for storing data to be analyzed, allowing users to independently accumulate data suitable for analysis in the storage device.

[0121] (Item 11) In a computer system described in any one of items 1 to 10, The aforementioned one or more computers are one computer. Computer system. This allows for faster analysis processing using a single computer, thus enabling more efficient use of resources. [Explanation of Symbols]

[0122] 10, 10a~10c Server, 12 Detection processing unit, 13 Clipping processing unit, 14, 14a Analysis processing unit, 20 Front-end interface, 21 Processor, 22 Switch, 23 Memory, 26 Back-end interface, 31 Compression unit, 32 RAG construction unit, 41 Encoder, 42 Decoder, 51 Video compression microservice, 52 Analysis processing microservice, 53 Feature generation microservice, 54 Vector database microservice, 71 Patch splitting processing unit, 72 Transformer, 91 Feature generation unit, 92 Vector database construction unit, 100, 100a~100c Computer system, 101 Object type specification interface, 110 Data source, 120 Storage device, 201 Management terminal.

Claims

1. A computing system having one or more computers and analyzing data, The aforementioned computer is Memory for storing programs, A processor that performs processing according to the aforementioned program, The processor executes the program, When the aforementioned data is input, a detection process is performed to detect a portion of the data that satisfies a condition specified by the user, and to generate first data indicating the position of the portion in the data. A cropping process to generate second data by cropping out the portion shown by the first data from the aforementioned data, The analysis process described above for analyzing the second data is performed. Computer system.

2. In the computer system described in claim 1, The aforementioned processor, If the data is video or still image data, the detection process converts the image data into first data which is binary data consisting of a first value indicating a pixel representing an object type specified by the user and a second value indicating a pixel representing an area other than the object type. As part of the cropping process, the second data, which includes a region indicating the first value, is cropped from the image data by referring to the first data. Computer system.

3. In the computer system described in claim 2, The aforementioned processor, If the image data is tensor data, the cropping process involves cropping a region from the image data, for each dimension of the height and width of the tensor data, where the minimum coordinate value of the first value pixel is the lower limit and the maximum coordinate value of the first value pixel is the upper limit, thereby generating the second data. Computer system.

4. In the computer system described in claim 2, The aforementioned processor, If the image data is tensor data, as a preliminary step to the analysis process, the image data is divided into a set of tokens, As part of the aforementioned cropping process, tokens corresponding to patches containing one or more pixels of the first value are cropped from the set of tokens. As a subsequent step in the analysis process, the set of extracted tokens is input into a neural network to obtain the second data from the neural network. Computer system.

5. In the computer system described in claim 4, The aforementioned processor, In addition to the tokens in the set of tokens that correspond to a patch containing one or more pixels of the first value, The aforementioned neural network extracts tokens at the location it is focusing on. Computer system.

6. In the computer system described in claim 2, The storage device has a storage device for storing the aforementioned image data, The aforementioned processor, As part of the analysis process, embeddings including the characteristics of the object type are obtained from the second data. A vector database is constructed in the storage device, with the aforementioned embedding as the key and the identifier of the image data as the value, associating the key with the value. Computer system.

7. In the computer system described in claim 6, The aforementioned processor, Based on the first data which is the detection result of the detection process, a compression process is performed to compress the image data by increasing the image quality of the region of the object type and decreasing the image quality of the region other than the object type, and then the compressed image data is associated with the detection result and stored in the storage device. Using the detection results, a decompression process is performed to decompress the compressed image data. The cropping process is performed using the expanded image data and the detection results. The analysis process is performed on the second data. Computer system.

8. In the computer system described in claim 2, The memory stores management information including a support set which is a pair of sample images in which the object type to be detected by a deep learning model for classifying images is depicted, and mask data indicating the region in the sample image in which the object type is depicted. The aforementioned processor, As part of the detection process, the management information and the image data are input to the deep learning model, and the first data is obtained from the deep learning model. Computer system.

9. In the computer system described in claim 8, Having an interface for setting the aforementioned management information, Computer system.

10. The computer according to any one of claims 1 to 9, A storage device for storing the aforementioned data, A computer system having [a certain feature].

11. In the computer system described in claim 1, The aforementioned one or more computers are one computer. Computer system.

12. A data processing method performed by a computer system, When data is input, the system detects the portion of the data that satisfies the conditions specified by the user, and generates first data indicating the position of the portion in the data. The portion shown by the first data is extracted from the aforementioned data to generate the second data. Analyzing the second data mentioned above, Data processing method.

13. On the computer, When data is input, the system detects the portion of the data that satisfies the conditions specified by the user, and generates first data indicating the position of the portion in the data. The portion shown by the first data is extracted from the aforementioned data to generate the second data. Analyzing the second data mentioned above, A program to perform a task.

Images