A device and computing system with integrated computing and storage architecture

Abstract

The present application discloses a device and a computing system with an integrated computing and storage architecture. By applying the technical solution of the present application, when it is detected that the computing device is about to receive a private data consisting of characters, in order to avoid being found by malicious users, the computing device may not directly receive and store the private data, but first convert the character data into audio data consisting of the voice of the storage person and image data consisting of the handwriting of the storage person. Only the private data that does not contain the character data is stored in the computing device, so that the data can be transmitted when a user with the same voice and handwriting is detected to retrieve the private data later.

Description

Technical Field

[0001] The present application relates to data management technology, and in particular to a device and a computing system with an integrated computing and storage architecture. Background Art

[0002] In related technologies, computing devices with integrated computing and storage architecture refer to computing devices that use a computing system that integrates network, computing, storage, and application core capabilities to manage data for business platforms.

[0003] Among them, the computing system with integrated computing and storage architecture in the related technology usually directly receives and stores the data that needs to be managed, which also makes it easy for the computing system or computing equipment to be attacked by malicious users, resulting in the leakage of important data. Summary of the invention

[0004] The embodiment of the present application provides a device and a computing system with an integrated computing and storage architecture, thereby solving the problem in the related art that data stored in a computing system with an integrated computing and storage architecture can be easily stolen by malicious users.

[0005] According to one aspect of an embodiment of the present application, a computing system with an integrated computing and storage architecture is provided, including:

[0006] When detecting that the computing device is about to receive the private character data of the user to be stored, acquiring the authenticated attribute information of the user to be stored, wherein the authenticated attribute information includes the first audio feature information and the first character feature information of the user to be stored;

[0007] Based on the first audio feature information and the first character feature information of the user to be stored, the private character data is converted by operation to obtain a first audio sub-part and a first image sub-part;

[0008] The first audio sub-portion and the first image sub-portion are stored, and upon receiving an extraction instruction of a user to be extracted, attribute information to be authenticated of the user to be extracted is obtained, wherein the attribute information to be authenticated includes second audio feature information and second character feature information of the user to be extracted;

[0009] Based on the second audio feature information and the second character feature information, the first audio sub-portion and the first image sub-portion are checked by calculation, and after the check passes, the first audio sub-portion and the first image sub-portion are sent to the user to be extracted.

[0010] Optionally, in another embodiment of the method of the present application, based on the first audio feature information and the first character feature information of the user to be stored, the private character data is converted by operation to obtain a first audio sub-part and a first image sub-part, including:

[0011] Splitting the private character data into a first part and a second part;

[0012] The first part is converted by operation to obtain the first audio sub-part composed of the first audio feature information; and the second part is converted by operation to obtain the first image sub-part composed of the first character feature information.

[0013] Optionally, in another embodiment of the method of the present application, performing operation conversion on the first part to obtain the first audio sub-part composed of the first audio feature information includes:

[0014] Inputting the first audio feature information into the audio encoder to obtain an audio embedding feature corresponding to the first audio feature information;

[0015] Inputting each audio embedding feature into a mapping layer in an audio codec deployed in a computing system, so that the mapping layer maps the audio embedding feature into a plurality of audio prefix vectors;

[0016] Input the plurality of audio prefix vectors into a speech construction model in the audio codec, so that the speech construction model performs audio description on the plurality of audio prefix vectors to obtain a reconstructed audio sub-portion;

[0017] Calculating a cross entropy loss between the reconstructed audio sub-portion and the first audio feature information, and performing gradient back propagation until the cross entropy loss is minimized, thereby determining to obtain the first audio sub-portion;

[0018] The cross entropy loss is determined by the following formula:

[0019] ;

[0020] in, is the audio prefix vector, and , X is the word embedding feature corresponding to the first audio feature information, and ;

[0021] Among them, the is the mapping layer in the audio codec, Audio embedding features.

[0022] Optionally, in another embodiment of the method of the present application, performing a computational transformation on the second portion to obtain the first image sub-portion composed of the first character feature information includes:

[0023] Acquire each character data constituting the second part; and generate a corresponding character image for each character data;

[0024] The character images are spliced ​​into the same image to obtain the first image sub-part.

[0025] Optionally, in another embodiment of the method of the present application, storing the first audio sub-portion and the first image sub-portion includes:

[0026] The first image sub-part is stored in the own device; and the first audio sub-part is stored in an industrial Internet of Things device associated with the own device.

[0027] Optionally, in another embodiment of the method of the present application, the step of obtaining the attribute information to be authenticated of the user to be extracted includes:

[0028] extracting a portion of audio content in the first audio sub-portion; and extracting a portion of character content in the first image sub-portion;

[0029] Sending a verification instruction to the user to be extracted, wherein the verification instruction is used to instruct the user to be extracted to generate second audio feature information based on the partial audio content and to generate second character feature information based on the partial character content;

[0030] The second audio feature information and the second character feature information are combined as the attribute information to be authenticated.

[0031] Optionally, in another embodiment of the method of the present application, after combining the second audio feature information and the second character feature information as the attribute information to be authenticated, the method further includes:

[0032] Calculating a first similarity between the first audio feature information and the second audio feature information; and calculating a second similarity between the first character feature information and the second character feature information;

[0033] After determining that both the first similarity and the second similarity exceed a preset qualified value, determining that the verification is passed;

[0034] The first similarity is calculated by the following formula:

[0035] P = a*Tn / (S1-S2);

[0036] Among them, P represents the value of the first similarity, a is a preset audio adjustment gradient, Tn is the duration of the first audio sub-part, and (S1-S2) is the vector difference information between the first audio feature information and the second audio feature information.

[0037] Optionally, in another embodiment of the method of the present application, sending the first audio sub-portion and the first image sub-portion to the user to be extracted includes:

[0038] converting the first audio sub-portion into corresponding character data and sending the data to the user to be extracted; and

[0039] Inputting the first image sub-portion into an image classification network to obtain a superpixel image corresponding to the first image sub-portion;

[0040] Performing a regional pooling operation on the superpixel image to determine a plurality of regional feature images contained in the superpixel image; and respectively calculating the overlap between each feature region image and the first image sub-portion to determine a target region feature image whose overlap with the feature region image exceeds a preset threshold;

[0041] Performing average pooling on each target feature region image to calculate the centroid value of the target feature region image;

[0042] Perform matrix multiplication and sum operation on the centroid value of the first image sub-portion and the centroid value of each target feature area image to obtain a feature sum value;

[0043] Based on the features and values, character images corresponding to each target feature region image are obtained, and each character image is converted into corresponding character data and then sent to the user to be extracted.

[0044] Optionally, in another embodiment based on the above method of the present application, the center of gravity value is calculated by the following formula:

[0045] .

[0046] Wherein, F is the centroid value of the first image sub-portion, and the value of 1() is: if a pixel point of the first image sub-portion belongs to the nth target feature area image, the value is 1, otherwise the value is 0.

[0047] According to another aspect of the embodiment of the present application, a computing device with an integrated computing and storage architecture is provided, including:

[0048] The detection module is configured to obtain the authenticated attribute information of the user to be stored when detecting that the computing device is about to receive the private character data of the user to be stored, wherein the authenticated attribute information includes the first audio feature information and the first character feature information of the user to be stored;

[0049] a conversion module configured to perform calculation conversion on the private character data based on the first audio feature information and the first character feature information of the user to be stored, so as to obtain a first audio sub-portion and a first image sub-portion;

[0050] an acquisition module, configured to store the first audio sub-portion and the first image sub-portion, and upon receiving an extraction instruction of a user to be extracted, acquire attribute information to be authenticated of the user to be extracted, wherein the attribute information to be authenticated includes second audio feature information and second character feature information of the user to be extracted;

[0051] The sending module is configured to perform an operation check on the first audio sub-part and the first image sub-part based on the second audio feature information and the second character feature information, and after the check is passed, send the first audio sub-part and the first image sub-part to the user to be extracted.

[0052] According to another aspect of the embodiments of the present application, a storage medium is provided, including:

[0053] A memory for storing executable instructions; and

[0054] The display is used to execute the executable instructions with the memory to complete the operation of any of the above-mentioned computing and storage integrated architecture computing systems.

[0055] According to another aspect of an embodiment of the present application, a computer-readable storage medium is provided for storing computer-readable instructions, which, when executed, perform the operations of any of the above-mentioned computing systems with an integrated computing and storage architecture.

[0056] In the present application, when it is detected that the computing device is about to receive the private character data of the user to be stored, the authenticated attribute information of the user to be stored is obtained, wherein the authenticated attribute information includes the first audio feature information and the first character feature information of the user to be stored; based on the first audio feature information and the first character feature information of the user to be stored, the private character data is converted by operation to obtain the first audio sub-portion and the first image sub-portion; the first audio sub-portion and the first image sub-portion are stored, and when an extraction instruction of the user to be extracted is received, the authenticated attribute information of the user to be extracted is obtained, wherein the authenticated attribute information includes the second audio feature information and the second character feature information of the user to be extracted; based on the second audio feature information and the second character feature information, the first audio sub-portion and the first image sub-portion are verified by operation, and after the verification passes, the first audio sub-portion and the first image sub-portion are sent to the user to be extracted.

[0057] By applying the technical solution of the present application, when it is detected that a computing device is about to receive certain private data consisting of characters, in order to prevent malicious users from finding the specified character data, the computing device may not directly receive and store the data, but first convert the character data into audio data consisting of the voice of the storage person and image data consisting of the handwriting of the storage person. Only the private data that does not contain the character data is stored in the computing device, so that the data can be transmitted when a user with the same voice and handwriting is detected to retrieve the private data later.

[0058] This reduces the problem in the related art that private data is easily stolen by malicious users due to the relatively simple existence of character data. On the other hand, by converting audio data and image data associated with the storage person, it also increases the difficulty for malicious users to retrieve private data in their name.

[0059] The technical solution of the present application is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The accompanying drawings, which constitute a part of the specification, illustrate embodiments of the present application and, together with the description, serve to explain the principles of the present application.

[0061] The present application can be more clearly understood from the following detailed description with reference to the accompanying drawings, in which:

[0062] Figure 1 A schematic diagram of a computing system with an integrated computing and storage architecture proposed in this application;

[0063] Figure 2 A flowchart of a computing system with an integrated computing and storage architecture proposed in this application;

[0064] Figure 3 A schematic diagram of the structure of a computing device with an integrated computing and storage architecture proposed in this application;

[0065] Figure 4 This is a schematic diagram of the structure of the storage medium proposed in this application. DETAILED DESCRIPTION

[0066] Various exemplary embodiments of the present application will now be described in detail with reference to the accompanying drawings. It should be noted that unless otherwise specifically stated, the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present application.

[0067] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0068] The following description of at least one exemplary embodiment is merely illustrative in nature and is not intended to limit the present application, its application, or uses.

[0069] Technologies, devices and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, devices and equipment should be considered as part of the specification.

[0070] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0071] In addition, the technical solutions between the various embodiments of the present application can be combined with each other, but it must be based on the fact that ordinary technicians in the field can implement it. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0072] It should be noted that all directional indications in the embodiments of the present application (such as up, down, left, right, front, back, etc.) are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0073] Combine the following Figure 1-Figure 2 To describe a computing system for performing a computing and storage integrated architecture according to an exemplary embodiment of the present application. It should be noted that the following application scenarios are only shown to facilitate understanding of the spirit and principles of the present application, and the embodiments of the present application are not limited in this regard. On the contrary, the embodiments of the present application can be applied to any applicable scenario.

[0074] The present application also proposes a device and a computing system with an integrated computing and storage architecture.

[0075] Figure 1 The flowchart of a computing system with an integrated computing and storage architecture according to an embodiment of the present application is schematically shown. Figure 1 As shown, including:

[0076] S101, when detecting that a computing device is about to receive private character data of a user to be stored, obtaining authenticated attribute information of the user to be stored, wherein the authenticated attribute information includes first audio feature information and first character feature information of the user to be stored.

[0077] S102: Based on the first audio feature information and the first character feature information of the user to be stored, the private character data is converted by operation to obtain a first audio sub-part and a first image sub-part.

[0078] S103, storing the first audio sub-portion and the first image sub-portion, and upon receiving an extraction instruction of the user to be extracted, obtaining attribute information to be authenticated of the user to be extracted, wherein the attribute information to be authenticated includes second audio feature information and second character feature information of the user to be extracted.

[0079] S104, based on the second audio feature information and the second character feature information, perform computational verification on the first audio sub-portion and the first image sub-portion, and after the verification passes, send the first audio sub-portion and the first image sub-portion to the user to be extracted.

[0080] In related technologies, computing devices with integrated computing and storage architecture refer to computing devices that use a computing system that integrates network, computing, storage, and application core capabilities to manage data for business platforms.

[0081] Specifically, computing devices with integrated computing and storage architecture can integrate server, storage, network, software and other configurations through standardized computing system architecture. This can not only reduce the complexity of data center infrastructure deployment and operation and maintenance management, but also solve the problems of continuous expansion of infrastructure in the big data era, the personalized and integrated needs of data processing, and the storage cost of massive data.

[0082] In related technologies, in order to prevent the leakage of private data stored in computing devices with integrated computing and storage architecture, business personnel usually manage data by having the computing system regularly encrypt private character data and uniformly store it in its own storage unit with higher security.

[0083] However, this data management method is relatively common, and it is also prone to the risk of private information being stolen due to the computing system being attacked by malicious users who are skilled in character data restoration.

[0084] Based on this, the present application proposes a computing system with an integrated computing and storage architecture, which can detect that the computing device is about to receive a private data consisting of characters. In order to avoid malicious users from finding the specified character data, the computing device can not directly receive and store it, but first convert the character data into audio data consisting of the voice of the storage person and image data consisting of the handwriting of the storage person. Only the private data that does not contain the character data is stored in the computing device, so that the data can be transmitted when a user with the same voice and handwriting is detected to retrieve the private data later.

[0085] Further, this application is hereby incorporated by reference Figure 2 Describe the plan in detail:

[0086] Step 1: When detecting that the computing device is about to receive the private character data of the user to be stored, the authenticated attribute information of the user to be stored is obtained.

[0087] The authenticated attribute information includes the first audio feature information and the first character feature information of the user to be stored.

[0088] In one embodiment, the private data is data composed of characters, wherein the characters may include text data, letter data, and numeric data, etc.

[0089] For example, when a certain private data is "the profit of Company A is 10 million", the data is private character data.

[0090] In one method, the authenticated attribute information of the user to be stored includes audio feature information corresponding to the human voice of a certain user (for example, the pitch feature, timbre feature, frequency feature, etc. of the user Zhang San). It can be understood that the present application requires the computing system to subsequently use the human voice of the user (i.e., the user to be stored) to generate corresponding audio data for it.

[0091] In another way, the authenticated attribute information of the user to be stored also includes character feature information corresponding to the handwriting of a certain user (for example, the font trajectory feature, font size feature, font thickness feature, etc. of the user Zhang San or Li Si). It can be understood that the present application requires the computing system to subsequently use the handwriting of the user (i.e., the user to be stored) to generate corresponding image data for it.

[0092] Step 2: Split the private character data into the first part and the second part.

[0093] In one embodiment, the present application does not limit the specific number of characters in the first part and the second part. For example, the two may be the same or different.

[0094] For example, when the private character data is "the profit of Company A is 10 million", the first part may be "the profit of Company A" and the second part may be "is 10 million".

[0095] Step 3: Performing computational transformation on the first part to obtain a first audio sub-part composed of first audio feature information; and performing computational transformation on the second part to obtain a first image sub-part composed of first character feature information.

[0096] In one way, due to the simplicity of the structure of character data, the present application chooses not to store character data in the computing device. Instead, it is converted into partial audio data and partial image data. In this way, even if it is subsequently invaded by a malicious user, the user's privacy can be prevented from being leaked because the corresponding character data cannot be found.

[0097] Furthermore, in order to avoid the problem of malicious users impersonating the user to be stored to retrieve private data from the computing device, the present application selects the voice of the user to be stored to generate the audio data during the process of generating the audio data. That is to say, when other users subsequently impersonate to retrieve the private data, the computing system can also refuse to provide the private data because it detects the inconsistency of the audio features of the two.

[0098] In one embodiment, the number of users to be stored may be one or more. It is understandable that when the number of users to be stored is more than one, it is possible to better ensure that the subsequent private data is difficult to be retrieved by fraudulent use.

[0099] In the process of obtaining the first audio sub-part, the embodiment of the present application may implement the following steps:

[0100] Inputting the first audio feature information into an audio codec deployed in the computing system to obtain an audio embedding feature corresponding to the first audio feature information;

[0101] Inputting each audio embedding feature into a mapping layer in an audio codec deployed in a computing system, so that the mapping layer maps the audio embedding feature into a plurality of audio prefix vectors;

[0102] Inputting the plurality of audio prefix vectors into a speech construction model in an audio codec, so that the speech construction model performs audio description on the plurality of audio prefix vectors to obtain a reconstructed audio sub-portion;

[0103] A cross entropy loss between the reconstructed audio sub-portion and the first audio feature information is calculated, and gradient back propagation is performed until the cross entropy loss is minimized, thereby determining that the first audio sub-portion is obtained.

[0104] In one way, the embodiment of the present application needs to first deploy an audio codec in the computing system. As an example, the audio codec includes a mapping layer and a speech construction model. It is understandable that the present application can pre-train multiple audio samples to enable the speech construction model to obtain the ability to reconstruct audio data from a multimodal semantic space.

[0105] In one embodiment, the present application requires that each acoustic feature in the first audio feature information Audio encoder fed into the audio codec In order to obtain the audio embedding features .

[0106] Furthermore, the mapping layer of the audio encoder Used to embed audio features Mapped to multiple audio prefix vectors , the specific steps can be expressed by the following formula:

[0107] .

[0108] Among them, the audio prefix vector corresponding to each audio feature information The same dimension as the audio embedding feature, the combination of all audio prefix vectors is the audio prefix .

[0109] Furthermore, each audio prefix The speech that is fed into the audio codec builds a model so that the audio data (i.e., the audio sub-part) can be reconstructed therefrom.

[0110] Furthermore, after obtaining the reconstructed audio sub-portion, in order to ensure its accuracy, the embodiment of the present application can also calculate the cross-entropy loss between the reconstructed audio and the original human voice (i.e., the first audio feature information) at least once, and perform gradient back propagation until the cross-entropy loss is minimized, and then determine to obtain the final first audio sub-portion.

[0111] As an example, the expression of cross entropy loss is as follows:

[0112] .

[0113] Where X is the word embedding feature corresponding to the first audio feature information, and .

[0114] Among them, in the process of obtaining the first image sub - part in the embodiments of the present application, the following steps may be implemented:

[0115] Obtain each character data that makes up the second part; and, generate a corresponding character image for each character data;

[0116] Splice each character image onto the same image to obtain the first image sub - part.

[0117] For example, when the private character data is "The profit of Company A is 10 million", the first part can be "The profit of Company A", and the second part can be "10 million".

[0118] That is to say, the present application needs to first use the voice of a user to be stored to generate audio data with the voice content of "The profit of Company A is 10 million" (i.e., the first audio sub - part). And it is also necessary to write 3 character data (i.e., "1", "0", "million") in the handwriting of a user to be stored and generate a corresponding character image for each character data.

[0119] For example, the character image 1 contains the digital character "1", the character image 2 contains the digital character "0", and the character image 3 contains the Chinese character "million". It can be understood that after splicing the character image 1, the character image 2, and the character image 3 together, the first image sub - part is obtained.

[0120] Step 4: Generate pseudo - private character data and store the pseudo - private character data in its own device.

[0121] In one way, in order to prevent the problem that a malicious user fails to find the corresponding data in the computing device and then reversely traces other storage devices (such as Internet of Things devices associated with the computing device), the present application can generate a pseudo - private character data in the computing device that has the same data length and data attributes as the original private character data. And store it in its own device to achieve the purpose of confusing malicious users.

[0122] Step 5: Store the first audio sub - part and the first image sub - part in the industrial Internet of Things device.

[0123] In one way, the present application can avoid any contact between the storage module of the computing device and the private character data, but directly send the first audio sub - part and the first image sub - part to the industrial Internet of Things device near it. That is to say, only a pseudo - private character data and the association relationship between the first audio sub - part and the first image sub - part are recorded in the computing device. Thus, it can avoid the drawback that once the converted data comes into contact with the storage module of the computing device, it can be directly known by malicious users when malicious users always monitor the computing device.

[0124] In one embodiment, the industrial Internet of Things device in the embodiment of the present application is a device with a certain storage space and capable of wireless data communication with a computing device. For example, it may be a smart home device (smart speakers, smart home appliances, etc.) deployed in the indoor office area where the computing device is located, industrial smart devices (smart lamp poles, smart traffic lights, etc.) deployed in the outdoor road area, data acquisition equipment deployed indoors and outdoors (smart water meters, smart electricity meters, smoke sensors, etc.), Internet of Vehicles equipment, etc.

[0125] Step 6: upon receiving an extraction instruction from the user to be extracted, extracting part of the audio content in the first audio sub-portion; and extracting part of the character content in the first image sub-portion.

[0126] In order to reduce the complexity of subsequent verification operations of the computing device, the present application can only extract the audio content of any part of the first audio sub-part and the character content of any part of the first image sub-part for verification.

[0127] It should be noted that in the process of calculation verification, the present application does not generate any audio data or character data of any content by the user to be extracted. Instead, it must generate a certain audio data contained in the first audio sub-part and a certain character data contained in the first image sub-part. In this way, it is ensured as much as possible that the fit between the second attribute information to be authenticated of the user to be extracted and the first attribute information to be authenticated of the user to be stored pre-stored satisfies the qualified conditions. It also avoids the problem of malicious users forging verification data in advance due to the use of fixed sample audio data or sample audio data.

[0128] Step 7: Send a verification instruction to the user to be extracted, and combine the second audio feature information and the second character feature information as attribute information to be authenticated.

[0129] The verification instruction is used to instruct the user to be extracted to generate second audio feature information based on part of the audio content, and to generate second character feature information based on part of the character content.

[0130] For example, when the first audio sub-part contains audio data of 12 text contents (e.g., "Company A's profit is 10 million"), the embodiment of the present application can only extract the audio data of 3 text contents (e.g., "Company A") and display them to the user, so that the user can read them aloud. It can be understood that the content read aloud by the user is the second audio feature information.

[0131] Similarly, when the first image sub - part contains image data with 5 - character content (such as "10 million"), embodiments of the present application can extract only the image data of 1 - character content (such as "million") from it and display it to the user for the user to copy accordingly. It can be understood that the content copied by the user is the second character feature information.

[0132] Step 8, calculate the first similarity between the first audio feature information and the second audio feature information; and calculate the second similarity between the first character feature information and the second character feature information.

[0133] It can be understood that only after the voice feature and character feature of the user to be extracted match the stored audio feature and character feature respectively, the computing device will recognize the user to be extracted as the user to be stored associated with the private character data. Then, the corresponding first audio sub - part and first image sub - part can be extracted from the Internet of Things device, restored, and sent to the user to be extracted.

[0134] Optionally, the first similarity is calculated through the following formula:

[0135] P = a * Tn / (S1 - S2);

[0136] Where P represents the value of the first similarity, a is a preset audio adjustment gradient, Tn is the duration of the first audio sub - part, and (S1 - S2) is the vector difference information between the first audio feature information and the second audio feature information.

[0137] Step 9, after determining that both the first similarity and the second similarity exceed the preset qualified value, determine that the verification is passed, and send the first audio sub - part and the first image sub - part to the user to be extracted.

[0138] In one way, the present application can send the original private character data to the user to be extracted through the following steps:

[0139] Convert the first audio sub - part into corresponding character data and send it to the user to be extracted;

[0140] In one way, the present application can use audio - to - text technology to convert the first audio sub - part into corresponding character data and send it to the user to be extracted.

[0141] In another way, the present application can input the first image sub - part into an image classification network to obtain a super - pixel image corresponding to the first image sub - part;

[0142] Performing a regional pooling operation on the superpixel image to determine a plurality of regional feature images contained in the superpixel image; and respectively calculating the overlap between each feature regional image and the first image sub-portion, and determining a target regional feature image whose overlap corresponding to the feature regional image exceeds a preset threshold;

[0143] Each target feature area image is average pooled to calculate the centroid value of the target feature area image; the centroid value of the first image sub-part and the centroid value of each target feature area image are matrix multiplied and summed to obtain a feature sum value; based on the feature sum value, the character image corresponding to each target feature area image is obtained, and each character image is converted into corresponding character data and sent to the user to be extracted.

[0144] By applying the technical solution of the present application, when it is detected that a computing device is about to receive certain private data consisting of characters, in order to prevent malicious users from finding the specified character data, the computing device may not directly receive and store the data, but first convert the character data into audio data consisting of the voice of the storage person and image data consisting of the handwriting of the storage person. Only the private data that does not contain the character data is stored in the computing device, so that the data can be transmitted when a user with the same voice and handwriting is detected to retrieve the private data later.

[0145] This reduces the problem in the related art that private data is easily stolen by malicious users due to the relatively simple existence of character data. On the other hand, by converting audio data and image data associated with the storage person, it also increases the difficulty for malicious users to retrieve private data in their name.

[0146] Optionally, in another embodiment of the method of the present application, based on the first audio feature information and the first character feature information of the user to be stored, the private character data is converted by operation to obtain a first audio sub-part and a first image sub-part, including:

[0147] Splitting the private character data into a first part and a second part;

[0148] The first part is converted by operation to obtain the first audio sub-part composed of the first audio feature information; and the second part is converted by operation to obtain the first image sub-part composed of the first character feature information.

[0149] Optionally, in another embodiment of the method of the present application, performing operation conversion on the first part to obtain the first audio sub-part composed of the first audio feature information includes:

[0150] Inputting the first audio feature information into the audio encoder to obtain an audio embedding feature corresponding to the first audio feature information;

[0151] Inputting each audio embedding feature into a mapping layer in an audio codec deployed in a computing system, so that the mapping layer maps the audio embedding feature into a plurality of audio prefix vectors;

[0152] Input the plurality of audio prefix vectors into a speech construction model in the audio codec, so that the speech construction model performs audio description on the plurality of audio prefix vectors to obtain a reconstructed audio sub-portion;

[0153] A cross entropy loss between the reconstructed audio sub-portion and the first audio feature information is calculated, and gradient back propagation is performed until the cross entropy loss is minimized, thereby determining to obtain the first audio sub-portion.

[0154] Optionally, in another embodiment of the method of the present application, performing a computational transformation on the second portion to obtain the first image sub-portion composed of the first character feature information includes:

[0155] Acquire each character data constituting the second part; and generate a corresponding character image for each character data;

[0156] The character images are spliced ​​into the same image to obtain the first image sub-part.

[0157] Optionally, in another embodiment of the method of the present application, storing the first audio sub-portion and the first image sub-portion includes:

[0158] The first image sub-part is stored in the own device; and the first audio sub-part is stored in an industrial Internet of Things device associated with the own device.

[0159] Optionally, in another embodiment of the method of the present application, the step of obtaining the attribute information to be authenticated of the user to be extracted includes:

[0160] extracting a portion of audio content in the first audio sub-portion; and extracting a portion of character content in the first image sub-portion;

[0161] Sending a verification instruction to the user to be extracted, wherein the verification instruction is used to instruct the user to be extracted to generate second audio feature information based on the partial audio content and to generate second character feature information based on the partial character content;

[0162] The second audio feature information and the second character feature information are combined as the attribute information to be authenticated.

[0163] Optionally, in another embodiment of the method of the present application, after combining the second audio feature information and the second character feature information as the attribute information to be authenticated, the method further includes:

[0164] Calculating a first similarity between the first audio feature information and the second audio feature information; and calculating a second similarity between the first character feature information and the second character feature information;

[0165] After determining that both the first similarity and the second similarity exceed a preset qualified value, it is determined that the verification passes.

[0166] Optionally, in another embodiment of the method of the present application, sending the first audio sub-portion and the first image sub-portion to the user to be extracted includes:

[0167] converting the first audio sub-portion into corresponding character data and sending the data to the user to be extracted; and

[0168] Inputting the first image sub-portion into an image classification network to obtain a superpixel image corresponding to the first image sub-portion;

[0169] Performing a regional pooling operation on the superpixel image to determine a plurality of regional feature images contained in the superpixel image; and respectively calculating the overlap between each feature region image and the first image sub-portion to determine a target region feature image whose overlap with the feature region image exceeds a preset threshold;

[0170] Performing average pooling on each target feature region image to calculate the centroid value of the target feature region image;

[0171] Perform matrix multiplication and sum operation on the centroid value of the first image sub-portion and the centroid value of each target feature area image to obtain a feature sum value;

[0172] Based on the features and values, character images corresponding to each target feature region image are obtained, and each character image is converted into corresponding character data and then sent to the user to be extracted.

[0173] Optionally, in another embodiment based on the above method of the present application, the center of gravity value is calculated by the following formula:

[0174] .

[0175] Wherein, F is the centroid value of the first image sub-portion, and the value of 1() is: if a pixel point of the first image sub-portion belongs to the nth target feature area image, the value is 1, otherwise the value is 0.

[0176] Optionally, in another embodiment of the present application, Figure 3 As shown, the present application also provides a computing device with an integrated computing and storage architecture. It includes:

[0177] The detection module is configured to obtain the authenticated attribute information of the user to be stored when detecting that the computing device is about to receive the private character data of the user to be stored, wherein the authenticated attribute information includes the first audio feature information and the first character feature information of the user to be stored;

[0178] a conversion module configured to perform calculation conversion on the private character data based on the first audio feature information and the first character feature information of the user to be stored, so as to obtain a first audio sub-portion and a first image sub-portion;

[0179] an acquisition module, configured to store the first audio sub-portion and the first image sub-portion, and upon receiving an extraction instruction of a user to be extracted, acquire attribute information to be authenticated of the user to be extracted, wherein the attribute information to be authenticated includes second audio feature information and second character feature information of the user to be extracted;

[0180] The sending module is configured to perform an operation check on the first audio sub-part and the first image sub-part based on the second audio feature information and the second character feature information, and after the check is passed, send the first audio sub-part and the first image sub-part to the user to be extracted.

[0181] By applying the technical solution of the present application, when it is detected that a computing device is about to receive certain private data consisting of characters, in order to prevent malicious users from finding the specified character data, the computing device may not directly receive and store the data, but first convert the character data into audio data consisting of the voice of the storage person and image data consisting of the handwriting of the storage person. Only the private data that does not contain the character data is stored in the computing device, so that the data can be transmitted when a user with the same voice and handwriting is detected to retrieve the private data later.

[0182] Figure 4 4 is a logical structure diagram of a storage medium according to an exemplary embodiment. For example, the storage medium 400 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0183] Reference Figure 4 , the storage medium 400 may include one or more of the following components: a processor 401 and a memory 402 .

[0184] The processor 401 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 401 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 401 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 401 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 401 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0185] Those skilled in the art will understand that Figure 4 The structure shown in the figure does not constitute a limitation on the storage medium 400, and may include more or less components than those shown in the figure, or combine certain components, or adopt a different component arrangement.

[0186] In an exemplary embodiment, a non-temporary computer-readable storage medium including instructions is also provided, such as a memory 404 including instructions. The above instructions can be executed by the processor 420 of the storage medium 400 to complete the above-mentioned call method, the method comprising: when it is detected that the computing device is about to receive the private character data of the user to be stored, obtaining the authenticated attribute information of the user to be stored, wherein the authenticated attribute information includes the first audio feature information and the first character feature information of the user to be stored; based on the first audio feature information and the first character feature information of the user to be stored, performing operation conversion on the private character data to obtain the first audio sub-portion and the first image sub-portion; storing the first audio sub-portion and the first image sub-portion, and when receiving the extraction instruction of the user to be extracted, obtaining the to-be-extracted attribute information of the user to be extracted, wherein the to-be-extracted attribute information includes the second audio feature information and the second character feature information of the user to be extracted; based on the second audio feature information and the second character feature information, performing operation verification on the first audio sub-portion and the first image sub-portion, and after the verification passes, sending the first audio sub-portion and the first image sub-portion to the user to be extracted. Optionally, the above instructions can also be executed by the processor 420 of the storage medium 400 to complete other steps involved in the above exemplary embodiments. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0187] In an exemplary embodiment, an application / computer program product is also provided, comprising one or more instructions, which can be executed by the processor 420 of the storage medium 400 to complete the above-mentioned call method, the method comprising: when it is detected that the computing device is about to receive the private character data of the user to be stored, obtaining the authenticated attribute information of the user to be stored, wherein the authenticated attribute information includes the first audio feature information and the first character feature information of the user to be stored; based on the first audio feature information and the first character feature information of the user to be stored, performing operation conversion on the private character data to obtain the first audio sub-portion and the first image sub-portion; storing the first audio sub-portion and the first image sub-portion, and when receiving the extraction instruction of the user to be extracted, obtaining the to-be-extracted attribute information of the user to be extracted, wherein the to-be-extracted attribute information includes the second audio feature information and the second character feature information of the user to be extracted; based on the second audio feature information and the second character feature information, performing operation verification on the first audio sub-portion and the first image sub-portion, and after the verification passes, sending the first audio sub-portion and the first image sub-portion to the user to be extracted. Optionally, the above instructions may also be executed by the processor 420 of the storage medium 400 to complete other steps involved in the above exemplary embodiments.

[0188] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

Claims

1. A computing system with an integrated computing and storage architecture, characterized in that: include: When detecting that the computing device is about to receive the private character data of the user to be stored, acquiring the authenticated attribute information of the user to be stored, wherein the authenticated attribute information includes the first audio feature information and the first character feature information of the user to be stored; Based on the first audio feature information and the first character feature information of the user to be stored, the private character data is converted by operation to obtain a first audio sub-part and a first image sub-part; The first audio sub-portion and the first image sub-portion are stored, and upon receiving an extraction instruction of a user to be extracted, attribute information to be authenticated of the user to be extracted is obtained, wherein the attribute information to be authenticated includes second audio feature information and second character feature information of the user to be extracted; Based on the second audio feature information and the second character feature information, performing computation verification on the first audio sub-portion and the first image sub-portion, and after the verification passes, sending the first audio sub-portion and the first image sub-portion to the user to be extracted; The method of performing calculation conversion on the private character data based on the first audio feature information and the first character feature information of the user to be stored to obtain a first audio sub-part and a first image sub-part includes: Splitting the private character data into a first part and a second part; Performing a calculation conversion on the first part to obtain the first audio sub-part composed of the first audio feature information; and performing a calculation conversion on the second part to obtain the first image sub-part composed of the first character feature information; The first part is converted by operation to obtain the first audio sub-part composed of the first audio feature information, including: Inputting the first audio feature information into an audio codec deployed in a computing system to obtain an audio embedding feature corresponding to the first audio feature information; Inputting each audio embedding feature into a mapping layer in the audio codec, so that the mapping layer maps the audio embedding feature into a plurality of audio prefix vectors; Input the plurality of audio prefix vectors into a speech construction model in the audio codec, so that the speech construction model performs audio description on the plurality of audio prefix vectors to obtain a reconstructed audio sub-portion; Calculating a cross entropy loss between the reconstructed audio sub-portion and the first audio feature information, and performing gradient back propagation until the cross entropy loss is minimized, thereby determining to obtain the first audio sub-portion; The cross entropy loss is determined by the following formula: ； in, is the audio prefix vector, and , X is the word embedding feature corresponding to the first audio feature information, and ; Among them, the is the mapping layer in the audio codec, Audio embedding features.

2. The system according to claim 1, characterized in that The second part is converted by operation to obtain the first image sub-part composed of the first character feature information, including: Acquire each character data constituting the second part; and generate a corresponding character image for each character data; The character images are spliced ​​into the same image to obtain the first image sub-part.

3. The system according to claim 1, characterized in that Storing the first audio sub-portion and the first image sub-portion includes: storing the first image sub-portion in a own device; and storing the first audio sub-portion in an industrial Internet of Things device associated with the own device; or, Generate pseudo-privacy character data and store the pseudo-privacy character data in its own device; and store the first audio sub-part and the first image sub-part in the industrial Internet of Things device.

4. The system according to claim 1, characterized in that The step of obtaining the to-be-authenticated attribute information of the to-be-extracted user includes: extracting a portion of audio content in the first audio sub-portion; and extracting a portion of character content in the first image sub-portion; Sending a verification instruction to the user to be extracted, wherein the verification instruction is used to instruct the user to be extracted to generate second audio feature information based on the partial audio content and to generate second character feature information based on the partial character content; The second audio feature information and the second character feature information are combined as the attribute information to be authenticated.

5. The system according to claim 4, characterized in that After combining the second audio feature information and the second character feature information as the attribute information to be authenticated, the method further includes: Calculating a first similarity between the first audio feature information and the second audio feature information; and calculating a second similarity between the first character feature information and the second character feature information; After determining that both the first similarity and the second similarity exceed a preset qualified value, determining that the verification is passed; The first similarity is calculated by the following formula: P = a*Tn / (S1-S2); Among them, P represents the value of the first similarity, a is a preset audio adjustment gradient, Tn is the duration of the first audio sub-part, and (S1-S2) is the vector difference information between the first audio feature information and the second audio feature information.

6. The system according to claim 1, wherein: Sending the first audio sub-portion and the first image sub-portion to the user to be extracted includes: converting the first audio sub-portion into corresponding character data and sending the data to the user to be extracted; and Inputting the first image sub-portion into an image classification network to obtain a superpixel image corresponding to the first image sub-portion; Performing a regional pooling operation on the superpixel image to determine a plurality of regional feature images contained in the superpixel image; and respectively calculating the overlap between each feature region image and the first image sub-portion to determine a target region feature image whose overlap with the feature region image exceeds a preset threshold; Performing average pooling on each target feature region image to calculate the centroid value of the target feature region image; Perform matrix multiplication and sum operation on the centroid value of the first image sub-portion and the centroid value of each target feature area image to obtain a feature sum value; Based on the features and values, character images corresponding to each target feature region image are obtained, and each character image is converted into corresponding character data and then sent to the user to be extracted.

7. The system according to claim 6, characterized in that The center of gravity value is calculated by the following formula: ； Wherein, F is the centroid value of the first image sub-portion, and the value of 1() is: if a pixel point of the first image sub-portion belongs to the nth target feature area image, the value is 1, otherwise the value is 0.

Images