Processing systems, learning systems, processing methods, learning methods, and programs

Abstract

To achieve optimal data processing. [Solution] A processing system comprising a conversion unit that converts m data to be processed, obtained by dividing input data which is an image or sound, into a discrete token sequence that can be input to a first machine learning model, the discrete token sequence being an array of n discrete tokens, wherein the conversion unit outputs the discrete token sequence containing nk discrete tokens, which are obtained by excluding k discrete tokens of low importance from the n discrete tokens (m, n, and k are each natural numbers, n is mutually arbitrary with respect to m, and n>k).

Description

Technical Field

[0001] The present invention relates to a processing system, a learning system, a processing method, a learning method, and a program.

Background Art

[0002] Conventionally, input data such as images and sounds has been converted into discrete tokens, which are one-dimensional information suitable for input into machine learning models such as VLM (Vision and Language Model) and LLM (Large Language Model), and the discrete tokens are input into the machine learning model to obtain a processing result, or input into a decoder to obtain restored data. Research has been underway on this. Since the amount of data of discrete tokens is compressed compared to the input data, using this as an intermediate representation has the merit of being able to speed up or simplify the processing. This merit is very beneficial in scenarios where high-speed processing is required, such as in autonomous driving, or in scenarios where data with a large amount of data, such as videos, is stored in a database.

[0003] In relation to this, Non-Patent Document 1 describes that in the technology of converting an image into a discrete token sequence, by pre-learning to drop (exclude) a random number of discrete tokens from the discrete token sequence at the end of token encoding for each frame, subsequent tokens can be omitted as needed during processing, thereby reducing the load of information processing.

[0004] Further, Patent Document 1 describes an apparatus that divides a moving image into line blocks, captures them as block data, and dynamically sets the compression rate of each block data. The compression rate of block data with a later capture start order is set lower than that of block data with an earlier capture start order. However, this technology does not convert input data into discrete tokens suitable for input into a machine learning model.

Prior Art Documents

Patent Documents

[0005] [Patent Document 1] Japanese Patent Publication No. 2011-211617 [Non-patent literature]

[0006] [Non-Patent Document 1] “ELASTICTOK: ADAPTIVE TOKENIZATION FOR IMAGE AND VIDEO”, Wilson Yan et al., released on October 10, 2024 [Non-Patent Document 2] "Release of Japanese Vision Language Model heron-blip-v1," Turing Inc., published March 6, 2024, https: / / zenn.dev / turing_motors / articles / 00df893a5e17b6 [Overview of the project] [Problems that the invention aims to solve]

[0007] In the technology described in Non-Patent Document 1, the number of input data items to be processed is the same as the number of feature information items to be discretized. Therefore, it was sometimes difficult to appropriately adjust the amount of information in the discrete token sequence or to adjust the processing load by arbitrarily changing the compression ratio. Furthermore, in the technology described in Non-Patent Document 1, if a machine learning model is used for the conversion to discrete tokens, there is a concern that the learning efficiency may decrease during the training process. Thus, conventional technologies sometimes failed to achieve suitable data processing.

[0008] This invention has been made in consideration of these circumstances, and one of its objectives is to provide a processing system, a learning system, a processing method, a learning method, and a program that can realize suitable data processing. [Means for solving the problem]

[0009] One aspect of the present invention is a processing system comprising a conversion unit that converts m data to be processed, obtained by dividing input data which is an image or sound, into a discrete token sequence that can be input to a first machine learning model, the discrete token sequence being an array of n discrete tokens, wherein the conversion unit outputs the discrete token sequence containing nk discrete tokens, obtained by excluding k discrete tokens of low importance from the n discrete tokens (where m, n, and k are natural numbers, n is mutually arbitrary with respect to m, and n>k).

[0010] Another aspect of the present invention is a learning system comprising: a first trial unit that inputs m child learning data obtained by dividing learning data, which is an image or sound, into a second machine learning model and outputs n feature information along with ordering, converts each of the nk feature information obtained by removing k feature information in ascending order from the n feature information into discretized feature information and encodes it, and outputs a discrete token sequence in which nk discrete tokens are arranged; a second trial unit that converts the nk discrete tokens into discretized feature information and obtains reconstructed data output by inputting the discretized feature information into a third machine learning model (where m, n, and k are natural numbers, n is mutually arbitrary with respect to m, and n>k); and an adjustment unit that adjusts the second and third machine learning models so that the evaluation result of evaluating the reproducibility of the reconstructed data with respect to the learning data is good. [Effects of the Invention]

[0011] According to one aspect of the present invention, suitable data processing can be achieved. [Brief explanation of the drawing]

[0012] [Figure 1] This is a diagram showing the configuration of the processing system according to the first embodiment. [Figure 2] This is a diagram (part 1) illustrating the processing of the processing system according to the first embodiment. [Figure 3] This is a diagram (part 2) illustrating the processing of the processing system according to the first embodiment. [Figure 4] It is a configuration diagram of a learning system. [Figure 5] It is a diagram (part 1) for explaining the processing of the learning system. [Figure 6] It is a diagram (part 2) for explaining the processing of the learning system. [Figure 7] It is a diagram showing an example in which a processing system is applied to an autonomous vehicle. [Figure 8] It is a diagram for explaining the processing of the processing system according to the second embodiment. [Figure 9] It is a diagram showing an example in which a processing system is applied to a data server. [Figure 10] It is a diagram for explaining the processing in the learning stage when dealing with voice. [Figure 11] It is a diagram for explaining the processing of the processing system according to the third embodiment.

Mode for Carrying Out the Invention

[0013] [Overview] The following describes embodiments of the processing system, learning system, processing method, learning method, and program of the present invention with reference to the drawings. Each of the processing system and the learning system is implemented, for example, by one or more processors. These may be dedicated systems or programs installed and functioning on a cloud server. The processing system converts the data to be processed, which is divided from input data such as images or audio, into a discrete token sequence that can be input into a first machine learning model such as VLM or LLM. The processing system reduces the processing load on subsequent processing by excluding some discrete tokens that are deemed to be of low importance from the discrete token sequence. Furthermore, the number of discrete tokens n included in the discrete token sequence in the processing system is mutually arbitrary with respect to the number of data to be processed m. "Mutually arbitrary" means that m may be arbitrarily determined for a fixed number n, or m may be arbitrarily determined for a fixed number m, or both n and m may be arbitrarily determined. This allows the processing system to achieve more flexible data processing. The learning system learns the second machine learning model included in the above-mentioned conversion unit in order to enable the processor to function as a processing system. These will be described in order below.

[0014] <First Embodiment> [Processing System] The processing system 100 of the first embodiment will be described below. Figure 1 is a configuration diagram of the processing system according to the first embodiment. The processing system 100 includes, for example, an acquisition unit 110, a processing unit 120, a conversion unit 130, and a storage unit 170. The conversion unit 130 includes an encoder unit 132 and a discretization / encoding unit 134. The components other than the storage unit 170 are realized, for example, by a hardware processor such as a CPU (Central Processing Unit) executing a program (software). Some or all of these components are LSIs (Large Scale Integrations), ASICs (Application Specific Integrated Circuits), FPGAs (Field-Programmable Graphite Arrays), etc. It may be implemented by hardware (including circuitry), such as a gate array, GPU (Graphics Processing Unit), or SOC (System On Chip), or by the cooperation of software and hardware. The program may be stored in advance in a storage device (a storage device equipped with a non-transitory storage medium), such as an HDD (Hard Disk Drive) or a flash memory, or may be stored in a removable storage medium (a non-transitory storage medium), such as a DVD or a CD-ROM, and installed by mounting the storage medium on a drive device.

[0015] The storage unit 170 is a RAM (Random Access Memory), HDD, flash memory, etc. The second machine learning model 172, codebook 174, etc. are stored in the storage unit 170. The storage unit 170 may not be attached to the processing system 100, but may be a NAS (Network Attached Storage) accessible to the processing system 100 via a network.

[0016] FIG. 2 is a diagram (part 1) for explaining the processing of the processing system according to the first embodiment. The acquisition unit 110 acquires input data 300, which is an image or audio. The acquisition unit 110 may acquire the input data 300 by communication from an external device, may acquire the input data 300 from a storage medium mounted on a drive device provided in the processing system 100, or may acquire the input data 300 by reading the input data from the non-volatile area of the storage unit 170 and storing it in the main processing area such as a RAM. In the example of FIG. 2, the input data 300 is an image.

[0017] The processing unit 120 processes the input data 300 to generate the data to be processed 310. For example, the processing unit 120 may divide the input data 300, which is a single still image, into m images of arbitrary sizes, or extract m frame images at different times from the input data 300, which is a moving image. In addition, the processing unit 120 may perform various other processing necessary for image processing, such as excluding the outer frame area of ​​the input data 300. Here, m may be a fixed value, set arbitrarily, or changed dynamically.

[0018] The conversion unit 130 converts the data to be processed 310 (here, each of m images obtained by dividing a single still image into m parts) into a discrete token sequence that can be input to the first machine learning model (described later), which is a discrete token sequence in which multiple discrete tokens are arranged. At this time, for each data to be processed 310, the conversion unit 130 outputs a discrete token sequence containing nk discrete tokens, which are obtained by excluding k discrete tokens of low importance from the n discrete tokens obtained from the data to be processed. m, n, and k are all natural numbers, and n is mutually arbitrary with respect to m, and n>k holds. Note that the system may be configured such that n≠m.

[0019] Let me explain in more detail. The encoder unit 132 of the conversion unit 130 inputs m data points 310 to be processed into the second machine learning model 172 and converts them into n feature information points 320#1 to 320#n. Feature information can be, for example, Haar-Like feature information, HOG feature information, SIFT feature information, or feature information obtained by machine learning models such as CNN, VGG, or ResNet. For example, feature information has the form of a vector and can be rephrased as a feature vector.

[0020] The second machine learning model 172 is a model trained to output n feature information 320#1 to 320#n in an ordered manner. The second machine learning model 172 is trained to output the n feature information 320#1 to 320#n in an ordered manner, with higher importance being prioritized. In the n feature information 320#1 to 320#n, a smaller number after "#" indicates a higher order (higher importance).

[0021] The discretization and coding unit 134 converts each of the nk feature information 320#1 to 320#nk, obtained by excluding the k feature information 320#n-k+1 to 320#n in descending order from the n feature information 320#1 to 320#n, into nk discrete tokens 330#1 to 330#nk, and outputs them as a sequence of discrete tokens. A discrete token is a code or symbol that is represented by discrete values ​​or a finite number of symbols (i.e., not a continuous value assuming significant figures). In the figure, the numbers 14, 21, and 13 are examples of discrete tokens. The discretization and coding unit 134 converts feature information into discrete tokens using, for example, a codebook 174. The codebook 174 is information that describes the corresponding discrete token for each of the multiple feature information (in the same form as those output by the encoder unit 132, for example, vectors of the same number of dimensions). For example, the discretization and coding unit 134 extracts the feature information with the smallest distance index, such as cosine similarity, (i.e., the closest content) for each of the nk feature information items 320#1 to 320#nk from the codebook 174, and determines each of the discrete tokens corresponding to the extracted feature information as nk discrete tokens 330#1 to 330#nk. In other words, the discretization and coding unit 134 discretizes the continuous feature information by matching it to one of the finite number of feature information items registered in the codebook 174, and then codes the discretized feature information by converting it into labels (codes) such as 14, 21, and 13. The discretization and coding unit 134 outputs this as a discrete token sequence. Alternatively, the discretization and coding unit 134 may convert each of the n feature information items 320#1 to 320#n into n discrete tokens, and then exclude the k discrete tokens with the lowest order. The discretization and coding unit 134 is not limited to the method using the codebook 174, but may also perform discretization using methods such as a Lookup-Free Quantizer, which discretizes each element of the vector into binary values ​​of 1 or -1.

[0022] Here, the second machine learning model 172 is trained together with the third machine learning model so that it produces a good evaluation result when the reproducibility of the reconstructed data to the training data is evaluated by inputting the m child training data obtained by dividing the training data, which is image or audio, into the second machine learning model 172, and thereby converting the sequence of discrete tokens, which contains nk discrete tokens that encode nk discrete feature information, into discrete feature information, and inputting the discrete feature information into the third machine learning model. The process of converting discrete tokens into discrete feature information is performed, for example, by reverse lookup in codebook 174 (obtaining feature information corresponding to discrete tokens). Through this training, the feature information that is the basis of discrete tokens of low importance is output from the second machine learning model 172 in a lower order.

[0023] The processing system 100 may also input n latent tokens 315 to the encoder unit 132 along with the data to be processed 310. Figure 3 is a diagram (part 2) illustrating the processing of the processing system according to the first embodiment. In this case, the encoder unit 132 outputs feature information in which m data to be processed 310 are projected onto an n-dimensional latent space corresponding to the n latent tokens 315. The latent tokens 315 and the mask tokens, which will be described later, are learned together when the second machine learning model or the third machine learning model is trained. Whether or not to input the latent tokens 315 to the encoder unit 132 depends on the configuration (architecture) of the encoder unit 132, and either option is acceptable. If the latent tokens 315 are not input to the encoder unit 132, the encoder unit 132 includes the function of projecting m data to be processed onto an n-dimensional latent space. The same applies to the second and third embodiments.

[0024] According to the processing system 100 of the first embodiment described above, for each data to be processed, a discrete token sequence containing nk discrete tokens is output, which is obtained by excluding k discrete tokens of low importance from the n discrete tokens obtained from the data to be processed. This reduces the processing load for subsequent processing. Specific examples of subsequent processing will be explained later in [Application Example 1], [Application Example 2], etc. Furthermore, by making n and m mutually arbitrary (there is no need to make n match m), the amount of information in the discrete token sequence can be adjusted as needed, and the compression ratio can be arbitrarily changed to adjust the processing load. As a result, suitable data processing can be achieved.

[0025] The number of latent tokens n corresponds to the expressive power when discrete tokenization is performed. Therefore, if the number of discrete tokens output is the same, increasing the number of latent tokens allows for greater expressive power (the ability to retain high-resolution image information). In contrast, in the method of Non-Patent Literature 1, the number of input patch sizes (the number of data to be processed in this application), the number of latent tokens, and the number of feature information outputs are the same. Therefore, it is not possible to adjust the expressive power as needed, but in this embodiment, n is made mutually arbitrary with respect to m, so it is possible to arbitrarily adjust the relative sizes of these during training. Therefore, if you want to increase the expressive power, you can build the system in that way. Conversely, if you want to perform inference calculations faster, it is advantageous to reduce the number of latent tokens. Again, in this embodiment, the system can be arbitrarily built in this way as well. Thus, according to this embodiment, the degree of freedom in system design is increased, and the system can be given desired characteristics.

[0026] The processing system 100 may be configured, for example, as a cloud server and include multiple conversion units 130 corresponding to at least two of still images, videos, and audio, and may selectively operate the conversion unit 130 according to the type of input data specified by the user. In this case, the conversion unit 130 for still images is trained on a second machine learning model 172 using training data that is still images, the conversion unit 130 for videos is trained on a second machine learning model 172 using training data that is video, and the conversion unit 130 for audio is trained on a second machine learning model 172 using training data that is audio.

[0027] [Learning System] The learning system 200 for learning the processing system 100 according to the first embodiment will be described below. Figure 4 is a diagram of the configuration of the learning system. The learning system 200 includes, for example, a first trial unit 210, a second trial unit 220, an adjustment unit 230, a weighting unit 240, and a storage unit 270. Components other than the storage unit 270 are realized, for example, by a hardware processor such as a CPU executing a program (software). Some or all of these components may be realized by hardware (including circuitry) such as an LSI, ASIC, FPGA, GPU, or SOC, or by the cooperation of software and hardware. The program may be stored in advance in a storage device such as an HDD or flash memory (a storage device equipped with a non-transient storage medium), or it may be stored in a removable storage medium such as a DVD or CD-ROM (a non-transient storage medium) and installed when the storage medium is mounted on a drive device.

[0028] The memory unit 270 can be RAM, HDD, flash memory, etc. The memory unit 170 stores the second machine learning model 272 in the learning phase, the third machine learning model 274 in the learning phase, the codebook 174, etc. The codebook 174 is the same as the one used by the discretization and coding unit 134 of the processing system 100. Similar to the processing system 100, a method that does not use the codebook 174 may be used. The memory unit 270 is not attached to the learning system 200, and may be a NAS that the learning system 200 can access via a network.

[0029] Figure 5 is a diagram (part 1) illustrating the processing of the learning system. The first trial unit 210 inputs m child learning data 410 and n latent tokens 415, obtained by dividing the learning data 400 (which is either an image or audio), into the second machine learning model 272 in the learning stage. The model outputs n feature information along with ordering, and outputs a discrete token sequence of nk discrete tokens 430#1 to 430#nk, obtained by removing k feature information in ascending order from the n feature information. The content of this processing is the same as that of the encoder unit 132 and the discretization / coding unit 134 of the processing system 100. Similarly, it is optional whether or not to input n latent tokens 415 into the second machine learning model 272 in the learning stage.

[0030] The second trial unit 220 converts nk discrete tokens 430#1 to 430#nk into discretized feature information and outputs reconstructed data 420 by inputting the discretized feature information along with m mask tokens 417 into the third machine learning model 274 in the learning phase (see description of the second machine learning model 172). The second machine learning model 272 and the third machine learning model 274 have an encoder / decoder relationship. The m mask tokens 417 act to return the discretized feature information in the nk-dimensional latent space back to the original m-dimensional space. Whether or not to input m mask tokens 417 into the third machine learning model 274 is arbitrary, just as whether or not to input n latent tokens 415 into the second machine learning model 272 in the learning phase in the processing system 100.

[0031] The adjustment unit 230 compares the reconstructed data 420 with the training data 400 and adjusts the parameters of the second machine learning model 272 and the third machine learning model 274 so that the evaluation result of the reproducibility of the reconstructed data 420 with respect to the training data 400 is good. The evaluation result of reproducibility is, for example, the sum or average of the difference in gradation for each pixel, and the smaller this value, the better the evaluation result. Reproducibility may be evaluated using any method, not limited to this. For example, if the SSIM, which is an image evaluation index, is used as the evaluation result, the larger the value, the better the evaluation result.

[0032] In the evaluation of reproducibility by the adjustment unit 230, the weights set by the weighting unit 240 may be reflected. The weighting unit 240 extracts a first region in the image, which is the training data 400, in which a predetermined object is represented, and assigns different weights to the first region and a second region that is not the first region. For example, if the processing system 100 is provided to an autonomous vehicle (hereinafter simply referred to as "vehicle") and determines the vehicle's behavioral guidelines based on images taken of the vehicle's exterior, the weighting unit 240 assigns the largest weight to pedestrians, bicycles, oncoming vehicles, traffic lights, etc., followed by large weights to preceding vehicles and vehicles driving alongside, and small weights to background objects such as the road surface and buildings. Various methods for object extraction in images are already known and will not be described in detail here, but for example, template matching and methods using machine learning models may be used.

[0033] Here, the method for evaluating the reproducibility of the reconstructed data 420 to the training data 400 is not limited to simply directly comparing the training data 400 and the reconstructed data 420, but the following methods may be adopted. Figure 6 is a diagram (part 2) for explaining the processing of the learning system. In this method, the adjustment unit 230 calculates the GAN (Generative Adversarial Network) Loss as the first evaluation result and the Perceptual Loss as the second evaluation result. The adjustment unit 230 may, for example, calculate an integrated evaluation result by taking a weighted sum of the first evaluation result and the second evaluation result, or it may perform parameter adjustments based on the first evaluation result and parameter adjustments based on the second evaluation result separately.

[0034] The adjustment unit 230 inputs the training data 400 and the reconstructed data 420 to the discriminator network 232. The discriminator network 232 functions as a GAN together with the first trial unit 210 and the second trial unit 220, which are used as the generator network. In other words, the discriminator network 232 determines from the image alone whether the training data 400 or the reconstructed data 420 is the original data (i.e., the training data 400), and outputs the determination result. GANLoss returns a good evaluation result when the discriminator network 232 outputs the correct determination result, and also returns a good evaluation result when the second machine learning model 272 and the third machine learning model 274 in the training stage cause the discriminator network 232 to make a wrong determination. In this way, the adjustment unit 230 repeatedly improves (adjusts parameters) both the generator network and the discriminator network 232 competitively, adjusting the reconstructed data 420 to approach the training data 400.

[0035] Meanwhile, the adjustment unit 230 inputs the training data 400 and the reconstructed data 420 to the image recognition model 234, respectively. The image recognition model 234 is, for example, a pre-trained VGG network that performs hierarchical processing to reduce dimensionality compared to at least the input layer and has the function of discriminating objects based on the feature information of the intermediate layers (for example, discriminating vehicles, pedestrians, buildings, etc. based on the feature information of the third layer from the input layer). The adjustment unit 230 calculates a second evaluation result based on the result of comparing one or more intermediate layers of the same hierarchical level. The "result of comparing intermediate layers" refers to values ​​such as L1 loss and L2 loss.

[0036] GANLoss enhances the "realism of the output," while PerceptualLoss works to improve "visual quality and detail reproduction." For example, in super-resolution and style transfer tasks, GANLoss can be used to pursue realism while PerceptualLoss corrects visually unnatural parts, resulting in realistic and high-quality image generation.

[0037] When the above process is repeatedly performed on multiple training data 400, the final second machine learning model 272 at the training stage is used as the second machine learning model 172 in the processing system 100.

[0038] According to the learning system 200 described above, a second machine learning model 172, which functions as part of the processing system 100, can be suitably trained. The basic operation of the learning system is the same even when constructing the processing system in the form shown in Figure 3, or the processing systems of the second and third embodiments.

[0039] Furthermore, according to the learning system 200, in addition to making n mutually arbitrary with respect to m, the number of k can also be arbitrarily determined, which allows for efficient learning even when the number of latent tokens is small. For example, in the method of Non-Patent Literature 1, when the input patch size and the number of latent tokens are large compared to the number of output discrete tokens, many learning iterations are required to consolidate the information into the output discrete tokens. For example, if the former is 4096 and the latter is 8, the information needs to be consolidated into 0.2% of the discrete tokens. In contrast, in the method of this embodiment, by setting the number of latent tokens n to, for example, 256 and the number of output discrete tokens nk to 8, it is sufficient to consolidate the information into 3.1% of the discrete tokens. Therefore, the learning efficiency, especially in the initial stages of learning, is improved compared to the method of Non-Patent Literature 1, and the processing system can be built more quickly.

[0040] [Application Example 1] The following describes an example of the application of the processing system. In this example, any of the first to third embodiments may be used, but the first embodiment will be used here. Figure 7 shows an example in which the processing system is applied to an autonomous vehicle. The autonomous vehicle, vehicle 10, is equipped with, for example, a camera 20, a detection device 22, a sensor 24, a first control device 26, a communication device 28, a second control device 30, a controlled object 32, and a storage device 34, and performs autonomous driving control using at least partially the processing results of the processing system 100.

[0041] Camera 20 is mounted anywhere on the vehicle 10 and captures images of the scenery in front of, to the side of, and behind the vehicle 10. Detection device 22 includes radar, LIDAR (Light Detection And Ranging), ultrasonic sensors, etc., and detects objects present around the vehicle 10. Sensor 24 includes speed sensors, acceleration sensors, gyro sensors, steering angle sensors, or accelerator opening sensors, brake sensors, steering torque pressure sensors, etc., which are used when manual driving is performed.

[0042] The first control unit 26 includes a processing system 100. The first control unit 26 operates, for example, by a processor such as a CPU executing a program stored in a memory device. Some or all of the functions of the first control unit 26 may be implemented by hardware such as an LSI, ASIC, FPGA, GPU, or SOC, or by the cooperation of software and hardware. The first control unit 26 performs processing for integrated perception and decision-making in autonomous driving. The processing system 100, which functions as part of the first control unit 26, takes images captured by the camera 20 as input data 300, and transmits the discrete token sequence generated by the processing system 100 as described above to the VLM server 40 using the communication device 28, and inputs it to the first machine learning model 42 of the VLM server 40. The communication device 28 accesses the network NW via wireless communication and communicates with the VLM server 40. The network NW includes wireless base stations, WANs, LANs, the internet, etc. The first machine learning model 42, as shown in Non-Patent Document 2, for example, includes an LLM (Large Language Model) and is trained to recognize the environment in which the vehicle 10 is located when a discrete token sequence is input, and to output necessary action guidelines. Action guidelines include information such as instructions regarding the medium- to long-term actions of the vehicle 10 and the reasons for making those instructions. For example, it is information close to natural language such as, "The traffic light is red, so please stop," "There is a sharp curve ahead, so please slow down gradually," "We will be turning left ahead, so please use your turn signal," and "There is an intersection ahead, so please turn right."

[0043] The first machine learning model 42 may be mounted on the vehicle 10 rather than being located outside the vehicle 10. In this case, for example, the processing system 100 may store the first machine learning model 42, other programs and data, etc., for functioning as a VLM or LLM in the memory unit 170, and the VLM or LLM function may be incorporated as a function of the encoder unit 132 or as a function of another functional unit.

[0044] The second control device 30 controls the controlled object 32 with more specific details while referring to the map information 36 stored in the storage device 34. The controlled object 32 includes, for example, drive force output devices such as engines, motors, and transmissions, brakes, steering devices, and turn signals. The second control device 30, for example, in response to a guideline such as "There is a sharp curve ahead, so please slow down gradually," decelerates from XX [m] [m / s] before the curve. 2 The control method, which involves decelerating at a certain point, is determined, and a control instruction to achieve this is output to the controlled object 32. The controlled object 32 may include ECUs (Electronic Control Units) that individually control each device.

[0045] In this way, by applying the processing system 100 to an autonomous vehicle, the amount of information exchanged with the first machine learning model 42 is reduced while retaining important information and generating a discrete token sequence, thereby suppressing a decrease in the accuracy of processing in the first machine learning model 42, i.e., subsequent processing.

[0046] In a processing system 100 applied to an autonomous vehicle, the acquisition unit 110 may acquire external information such as speed and driving environment (external information related to input data), and the value of k may be dynamically changed based on the external information. For example, when the vehicle 10 is moving at a low speed, the amount of change in the image from the camera 20 is small, so even if the value of k is increased, it is considered that the effect on the driving control of the vehicle 10 will be small. Therefore, the conversion unit 130 may increase the value of k as the speed of the vehicle 10 decreases. Specifically, the conversion unit 130 may make the value of k when the speed of the vehicle 10 is below a threshold larger than the value of k when the speed of the vehicle 10 is above a threshold. Furthermore, the conversion unit 130 may change the value of k based on the attributes of the location where the vehicle 10 is traveling, either by analyzing the image captured by the camera 20 or based on a positioning device (not shown) and map information 36. For example, the conversion unit 130 may decrease the value of k when the vehicle 10 is traveling in an urban area because there is a large amount of information about the external environment, and increase the value of k when the vehicle 10 is traveling on a road for vehicles only because there is a small amount of information about the external environment. Furthermore, the conversion unit 130 may, based on the analysis of the image captured by the camera 20 or the output value of an illuminance sensor (not shown), reduce the value of k when the vehicle 10 is driving at night or in rainy or snowy conditions, as this requires acquiring more external information, and increase the value of k when the vehicle 10 is driving during the daytime. In this way, the processing system 100 can reduce the amount of unnecessary data processing and perform processing more quickly. In addition, the processing system 100 can maintain high processing accuracy by increasing the amount of data processing when necessary.

[0047] According to the first embodiment described above, it is possible to reduce the processing load on subsequent processing while suppressing a decrease in accuracy.

[0048] <Second Embodiment> The second embodiment will be described below. Since the basic operation of the components is the same as in the first embodiment, the same names and reference numerals will be used as in the first embodiment. Figure 8 is a diagram illustrating the processing of the processing system according to the second embodiment.

[0049] In the second embodiment, the processing unit 120 generates processing target data 310 by extracting multiple still images and timestamps from the input data 300, which is a moving image. In this case, the processing target data 310 is a sequence of multiple timestamps and still images. In this case, the interval between timestamps, i.e., the time interval between still images, does not have to be constant (for example, still images with timestamps t=1, 3, 8, ... may be extracted). A timestamp is information that represents the order and time of still images, such as the elapsed time when the start time in the video is set to zero. Also in this case, the latent tokens 315 input to the encoder unit 132 are also arranged in the time direction.

[0050] [Application Example 2] The following describes an application example 2 of the processing system. In application example 1, any of the first to third embodiments may be used, but here the second embodiment will be applied. The processing system is applied to a data server. The data server stores, for example, video footage distributed via a network NW in the form of a discrete token sequence. The video footage is restored based on the discrete tokens in another terminal device or server. The processing system is implemented by the data server and another terminal device or server having a restoration unit 140. Figure 9 shows an example of the processing system being applied to a data server. The restoration unit 140 receives discrete tokens 330#1 to 330#nk and m mask tokens 317 as input. The restoration unit 140 converts the discrete tokens 330#1 to 330#nk into discrete feature information using the codebook 174, inputs the discrete feature information together with the m mask tokens 417 into a third machine learning model 176, and obtains the output of the third machine learning model 176 to restore a video footage (restored data) that is close to the original video footage, which is the input data 300.

[0051] In Application Example 2, the value of k may be changed so that the image evaluation index, such as SSIM, calculated by comparing the original video input data 300 with the video output by the reconstruction unit 140, is equal to or greater than a predetermined value. The image evaluation index is an example of "an evaluation result that assesses the reproducibility of the reconstructed data with respect to the input data or the data to be processed." For example, the conversion unit 130 decreases the value of k if the image evaluation index is less than a predetermined value, and maintains or increases the value of k if it is equal to or greater than a predetermined value. The conversion unit 130 may use a dedicated machine learning model when deciding how much to change the value of k.

[0052] [If the input data is audio] As mentioned above, the input data may be audio. When the input data is audio, the processing unit 120 performs processing such as Mel spectrogram conversion and frequency filtering. Figure 10 is a diagram illustrating the processing during the learning phase when handling audio. The same reference numerals are used for the same functional units as in Figure 5. The first trial unit 210 inputs the audio divided into audio sections, the child learning data 410 including timestamps, and the latent tokens 415 into the second machine learning model 272 in the learning phase, causing it to output n feature information along with ordering, and then outputs a discrete token sequence of nk discrete tokens 430#1~430#nk, which are obtained by excluding k feature information in ascending order from the n feature information.

[0053] The second trial unit 220 performs the reverse process of discretization on nk discrete tokens 430#1 to 430#nk, and outputs reconstructed data 420 which is obtained by inputting the data processed using the mask token 417 into the third machine learning model 274 in the training phase.

[0054] The adjustment unit 230 compares the reconstructed data 420 with the training data 400 and adjusts the parameters of the second machine learning model 272 and the third machine learning model 274 so that the evaluation result of the reproducibility of the reconstructed data 420 with respect to the training data 400 is good. The evaluation result of reproducibility is, for example, the sum or average of the differences when comparing the intensity for each frequency, and the smaller this value, the better the evaluation result. The reproducibility may be evaluated using any method, not limited to this.

[0055] In the evaluation of reproducibility by the adjustment unit 230, the weights set by the weighting unit 240 may be reflected. The weighting unit 240 sets weights for frequencies based on the noise intensity of the speech, which is the training data 400, or on pre-provided objective information. The objective information is information that represents the use and purpose of the processing system 100. For example, if the noise intensity is high, the weighting unit 240 sets a small weight for the frequency band corresponding to the noise so that many tokens in that frequency band are reduced. Also, if the objective information represents "speech recognition," the weighting unit 240 sets a small weight for the frequency band corresponding to silence or ambient noise so that the speaker's utterances are preferentially tokenized. In this way, the weighting unit 240 may perform tokenization that emphasizes specific information (emotional characteristics, voice quality, content, etc.) depending on the purpose of use of the speech data.

[0056] Furthermore, in the process of changing the value of k in Application Example 2, the value of k may be dynamically changed depending on the audio scene and application. For example, if the target information mentioned above is "speech recognition," the conversion unit 130 may change the value of k based on context prediction. Also, if the target information is "audio streaming," the conversion unit 130 may change the value of k according to the bandwidth of the audio being streamed.

[0057] According to the second embodiment described above, similar to the first embodiment, it is possible to reduce the processing load of subsequent processing while suppressing a decrease in accuracy.

[0058] <Third Embodiment> The third embodiment will be described below. Since the basic operation of the components is the same as in the second embodiment, the same names and reference numerals will be used as in the second embodiment. Figure 11 is a diagram illustrating the processing of the processing system according to the third embodiment. In the third embodiment, first a reference frame image is input to the encoder unit 132. The reference frame image is the first frame image in a video, or the first few frames of a video. The target frame is an image that follows the reference frame image (a frame image later in time), and may be a difference image relative to the reference frame image.

[0059] According to the third embodiment described above, similar to the first embodiment, it is possible to reduce the processing load of subsequent processing while suppressing a decrease in accuracy.

[0060] Although embodiments for carrying out the present invention have been described above using examples, the present invention is not limited in any way to these embodiments, and various modifications and substitutions can be made without departing from the spirit of the present invention. [Explanation of Symbols]

[0061] 10 vehicles 40 VLM servers 42. First Machine Learning Model 100 Processing Systems 110 Acquisition Department 120 Processing Department 130 Conversion Unit 132 Encoder section 134 Discretization / encoding section 140 Restoration Section 172 Second Machine Learning Model 174 Codebooks 176 Third Machine Learning Model 200 Learning Systems 210 1st trial part 220 2nd trial part 230 Adjustment section 240 Weighting section

Claims

1. The system includes a conversion unit that converts m data points obtained by dividing input data, which is an image or sound, into a discrete token sequence that can be input to a first machine learning model, the discrete token sequence being an array of n discrete tokens. The conversion unit outputs a sequence of discrete tokens containing n-k discrete tokens, which are obtained by removing k discrete tokens of low importance from the n discrete tokens. (m, n, and k are natural numbers, n is mutually arbitrary with respect to m, and n > k) Processing system.

2. The conversion unit includes an encoder unit that inputs m data items to be processed into a second machine learning model and converts them into n feature information items. The second machine learning model is a model trained to output the n feature pieces in an ordered manner, The conversion unit further includes a discretization and encoding unit that converts each of the n-k pieces of feature information obtained by removing k pieces of the n pieces of feature information in order of decreasing order into discretized feature information, encodes them, and outputs n-k discrete tokens. The processing system according to claim 1.

3. The second machine learning model is trained together with the third machine learning model to obtain a good evaluation result when the reproducibility of the reconstructed data to the training data is evaluated by inputting the m child training data obtained by dividing the training data, which is an image or sound, into the second machine learning model, and thereby converting the sequence of discrete tokens, which contains n-k discrete tokens that encode n-k discrete feature information, into discrete feature information, and inputting the discrete feature information into the third machine learning model. The processing system according to claim 2.

4. The conversion unit changes the value of k based on external information related to the input data. The processing system according to claim 1.

5. The system further includes a reconstruction unit that converts the discrete token sequence output by the conversion unit into discrete feature information, and outputs reconstructed data by inputting the discrete feature information into a third machine learning model. The processing system according to claim 2.

6. The conversion unit changes the value of k based on the evaluation result of evaluating the reproducibility of the restored data output by the restoration unit with respect to the input data or the data to be processed. The processing system according to claim 5.

7. The system comprises multiple conversion units that correspond to at least two of still images, videos, and audio, The conversion unit is selectively operated according to the type of input data. The processing system according to claim 1.

8. A data acquisition unit that acquires input data, The system further comprises a processing unit that divides the input data to generate m pieces of the data to be processed, The aforementioned input data is a video image. The processing unit extracts m frame images from the moving image to generate the data to be processed. The time intervals between the m frame images are not constant. The processing system according to claim 1.

9. The aforementioned input data is a moving image of the external scenery captured by a moving object. The first machine learning model outputs action guide information for the autonomous movement of the mobile object when the discrete token sequence is input. The conversion unit outputs the discrete token sequence as data to be input to the first machine learning model. The processing system according to claim 1.

10. An encoder unit that divides input data, which is either an image or audio, into m data points to be processed, inputs them into a second machine learning model, and converts them into n ordered feature information. The system comprises a discretization and encoding unit that converts each of the n feature information into n discretized feature information and encodes them to convert them into n discrete tokens, and outputs a discrete token sequence containing n-k discrete tokens obtained by removing k discrete tokens from the n discrete tokens in descending order of the corresponding feature information, (where m, n, and k are natural numbers, n is mutually arbitrary with respect to m, and n > k) The second machine learning model is trained together with the third machine learning model to obtain a good evaluation result when the reproducibility of the reconstructed data to the training data is evaluated by inputting the m child training data obtained by dividing the training data, which is an image or sound, into the second machine learning model, and thereby converting the sequence of discrete tokens, which contains n-k discrete tokens that encode n-k discrete feature information, into discrete feature information, and inputting the discrete feature information into the third machine learning model. Processing system.

11. The input data, the restored data, and the training data are images. The evaluation results for assessing the reproducibility are derived based on both a first evaluation result obtained by directly comparing the reconstructed data and the training data, and a second evaluation result obtained by comparing the intermediate layers when the reconstructed data and the training data are input into an image recognition model that performs dimensionality reduction. The processing system according to claim 3 or 10.

12. The first evaluation result was GAN (Generative Adversarial Network) Loss, The second evaluation result is Perceptual Loss. The processing system according to claim 11.

13. The processing system according to claim 1 or 10, wherein n ≠ m.

14. A first trial unit inputs m child training data obtained by dividing training data, which is either an image or an audio, into a second machine learning model, outputs n feature information along with ordering, removes k feature information in ascending order from the n feature information, converts each of the n-k feature information obtained by removing them into discretized feature information, encodes them, and outputs a discrete token sequence in which n-k discrete tokens are arranged (where m, n, and k are natural numbers, n is mutually arbitrary with respect to m, and n > k). A second trial unit obtains reconstructed data by converting n-k discrete tokens into discrete feature information and inputting the discrete feature information into a third machine learning model, An adjustment unit that adjusts the second machine learning model and the third machine learning model so that the evaluation result of evaluating the reproducibility of the restored data with respect to the training data is good, A learning system equipped with the following features.

15. The aforementioned training data and the aforementioned reconstructed data are images. The evaluation results for assessing the reproducibility are derived based on both a first evaluation result obtained by directly comparing the training data and the reconstructed data, and a second evaluation result obtained by comparing the intermediate layers when the training data and the reconstructed data are input into an image recognition model that performs dimensionality reduction. The learning system according to claim 14.

16. The first evaluation result was GAN (Generative Adversarial Network) Loss, The second evaluation result is Perceptual Loss. The learning system according to claim 15.

17. The aforementioned training data is an image, The image further includes a weighting unit that extracts a first region in which a predetermined object is represented and assigns different weights to the first region and a second region that is not the first region. The adjustment unit evaluates the reproducibility by taking the weight into account. The learning system according to claim 14.

18. The aforementioned training data is audio. The system further includes a weighting unit that assigns different weights to frequency bands according to pre-given objective information. The adjustment unit evaluates the reproducibility by taking the weight into account. The learning system according to claim 14.

19. The learning system according to claim 14, wherein n ≠ m.

20. A method of processing performed by a computer, The system includes a process to convert m data points obtained by dividing input data, which is an image or audio, into a discrete token sequence that can be input to a first machine learning model, which is a discrete token sequence in which n discrete tokens are arranged. The conversion process includes outputting a sequence of discrete tokens containing n-k discrete tokens, which are obtained by excluding k discrete tokens of low importance from the n discrete tokens. (m, n, and k are natural numbers, n is mutually arbitrary with respect to m, and n > k) Processing method.

21. A learning method performed by a computer, The process involves dividing the training data, which is either an image or an audio, into m child training data points, inputting these into a second machine learning model, and outputting n feature information along with their ordering. The process involves removing k of the aforementioned feature information in ascending order from the n feature information items, thereby obtaining n-k feature information items, converting each of these n-k feature information items into discretized feature information, encoding it, and outputting a discrete token sequence in which n-k discrete tokens are arranged (where m, n, and k are natural numbers, n is mutually arbitrary with respect to m, and n > k). A process to obtain reconstructed data output by converting n-k discrete tokens into discrete feature information and inputting the discrete feature information into a third machine learning model, A process to adjust the second machine learning model and the third machine learning model so that the evaluation result of evaluating the reproducibility of the restored data with respect to the training data is good, A learning method that includes [the following features].

22. On the computer, A program for performing a process to convert m data points obtained by dividing input data, which is an image or audio, into a discrete token sequence that can be input to a first machine learning model, the discrete token sequence being an array of n discrete tokens, The conversion process includes outputting a sequence of discrete tokens containing n-k discrete tokens, which are obtained by excluding k discrete tokens of low importance from the n discrete tokens. (m, n, and k are natural numbers, n is mutually arbitrary with respect to m, and n > k) program.

23. On the computer, The process involves dividing the training data, which is either an image or an audio, into m child training data points, inputting these into a second machine learning model, and outputting n feature information along with their ordering. The process involves removing k of the aforementioned feature information in ascending order from the n feature information items, thereby obtaining n-k feature information items, converting each of these n-k feature information items into discretized feature information, encoding it, and outputting a discrete token sequence in which n-k discrete tokens are arranged (where m, n, and k are natural numbers, n is mutually arbitrary with respect to m, and n > k). A process to obtain reconstructed data output by converting n-k discrete tokens into discrete feature information and inputting the discrete feature information into a third machine learning model, A process to adjust the second machine learning model and the third machine learning model so that the evaluation result of evaluating the reproducibility of the restored data with respect to the training data is good, A program to execute.

Images