35results about How to "To achieve deduplication" patented technology

The invention discloses a data deduplication system with ownership management for cloud ciphertext storage and a data deduplication method thereof. The method includes the steps of installation, datauploading, file level ownership management, data block level ownership management, and data downloading. The scheme of the invention supports deduplication of cross-user file level data and deduplication of single-user data block level data, has security and high efficiency, and can resist pollution attacks and data forgery attacks for the deduplication of the cross-user file level data. In addition, a delayed update strategy in the invention can reduce the data update frequency of the deduplication of the cross-user file level data and reduce the computation overhead of the system while ensuring the forward and backward confidentiality; and a user-assisted convergence encryption method of the invention can implement the deduplication of the single-user data block level data.

The invention provides a method and system for processing an asynchronous task. The method and system are helpful to improve the performance of an asynchronous task processing system and meanwhile guarantee the reliability. The method comprises the steps that a gateway inserts just received task data to a task data queue of a Redis according to an algorithm; an asynchronous task processing program acquires the task data from a Redis queue according to a scheduling algorithm, and then relevant task processing is carried out according to the acquired task data; under the condition that the Redis is abnormal, a database task list is used for replacing the Redis to conduct task processing.

The invention discloses an outsourcing data deduplication cloud storage method supporting privacy and integrity protection. Firstly, client deduplication and security authentication label deduplication of a data file are realized, so that the communication overhead and the storage burden of a cloud server are greatly reduced; and secondly, by adopting the extractor technology, the ciphertext of the outsourcing storage data does not depend on an encipher and an encryption algorithm, but depends on the data, so that ciphertext data deduplication is realized, and the privacy of the data is ensured in the deduplication process and the integrity auditing process. And finally, the RS erasure code technology is adopted to improve the fault-tolerant capability of deduplication cloud storage; a cloud service provider is also supported to spontaneously and regularly delegate a third-party auditor to carry out batch public cloud auditing on the data stored in the cloud server so as to ensure theintegrity of the data; meanwhile, a data owner is also supported to initiate public cloud auditing on the integrity of the file anytime and anywhere, so that the privacy of the data can be always protected in auditing and deduplication processes. The method provided by the invention is effective in the aspect of deduplication and efficient in the aspect of integrity cloud auditing.

The invention provides monitoring video file generation and playing methods and devices and terminal equipment. The monitoring video file generation method comprises the steps of: acquiring a real-time video frame shot by a monitoring camera; inquiring a prior video frame consistent with the real-time video frame in a de-duplicated video frame set; if the prior video frame consistent with the real-time video frame is not inquired, storing the real-time video frame into the de-duplicated video frame set, and recording a mapping relationship between playing time and the real-time video frame ina playing index; if the prior video frame consistent with the real-time video frame is inquired, abandoning the real-time video frame, and recording a mapping relationship between the playing time andthe prior video frame in the playing index; and according to the de-duplicated video frame set and the playing index, generating a monitoring video file. According to the invention, a size of the monitoring video file can be reduced, and duration of a storable monitoring video is prolonged.

Embodiments of the invention disclose a data deduplication method, a data deduplication device, electronic equipment and a computer readable storage medium. The data deduplication method comprises thesteps of: acquiring a to-be-processed data container; searching a data container of which the data similarity with the to-be-processed data container meets a preset condition from existing data containers, and taking the data container as a target data container; and comparing the to-be-processed data container with the target data container, confirming duplicated data, and deleting the duplicated data in a post-processing flow. According to the data deduplication method and the data deduplication device, mass data deduplication in the global range can be achieved, and the purpose of saving storage space is achieved on the premise that the user IO performance is not reduced.

The invention discloses a reporting data processing method and device, terminal equipment and a server, and the method comprises the steps: obtaining a reporting instruction through at least one terminal, and obtaining a multimedia data segment from multimedia data played by a media playing interface according to the reporting instruction; The at least one terminal obtains a reporting submission instruction and generates reporting data according to the reporting submission instruction; At least one terminal obtains the reporting time and sends the reporting data and the reporting time to the server; The server receives the reporting data and the reporting time sent by at least one terminal, and obtains the current server time corresponding to each reporting data and reporting time; The server obtains the real reporting time of the at least one reporting instruction according to the at least one current server time and the corresponding reporting time; And the server generates audit data based on the at least one piece of report data with the same real report time. By calculating the real reporting time of the reporting data, duplicate removal of the reporting data is realized.

The invention discloses an image management method. The image management method comprises the steps of obtaining multiple images created in a predetermined time period and identifying repeated images in the images; according to a preset low-quality image category, judging whether the repeated images are low-quality images or not; when the repeated images are the low-quality images, determining quality values of the repeated images; and according to the quality values of the repeated images, determining to-be-deleted repeated images. The invention furthermore discloses an apparatus corresponding to the method. Image management is facilitated.

In order to overcome the defects in the prior art, the invention provides a Pattern file loading method in ATE equipment, which comprises the following steps of: introducing a redis memory database, and storing a Pattern file by using a data structure combined by two hash tables; when the Pattern file is loaded, the Pattern file is loaded into a memory through the data structure of the hash combination; the Pattern file is loaded into the FPGA from the memory through N threads; the data structure of the combination of the two hashes comprises a hash1 constructed by a key1, a hash2 constructed by a key1, and a hash3 constructed by a key2; and a hash2 is constructed by the key 2 and the value 2. The method has the advantages of simple data structure and small occupied memory; and meanwhile, the redis database is introduced, so that the same vector data in the pattern can be prevented from being repeatedly loaded by importing the analyzed database, and the memory consumption and the time consumption are reduced.

The invention discloses a file storage method. The method comprises the steps that a cloud searches whether a first hash value exists or not; if yes, the cloud returns prompt information of repeated files and waits for uploading of the second ciphertext; if not, the cloud requests for the to-be-uploaded file; and the client encrypts the to-be-uploaded file to obtain an encrypted file, encrypts thefirst preset key to obtain a first ciphertext, encrypts the second hash value by using a second preset key to obtain a second ciphertext, and uploads the first ciphertext, the second ciphertext and the encrypted file to the cloud. According to the method, the to-be-uploaded file is in a ciphertext form when leaving the client, the unsafe environment on the network and the cloud is isolated, meanwhile, the second hash value of the file is adopted to encrypt the first preset key to ensure that the key of the encrypted file is not leaked, data deduplication is carried out on the encrypted data,and meanwhile, the security of the data is ensured. The invention also provides a file storage system, cloud equipment and terminal equipment, which have the above beneficial effects.

The invention discloses a method and a system for deduplicating files of a cloud storage system. The method comprises the following steps: reading a file to be uploaded, and calculating the information abstract value of the file; comparing the information abstract value of the file with the information abstract values of files which have existed in the cloud storage system, and if the file of which the information abstract value is the same as those of the files exists in the cloud storage system, stopping uploading the file. According to the method and the system for deduplicating the files of the cloud storage system disclosed by the invention, the information abstract value of the file is used, the information abstract value of the file is of uniqueness, and if the file of which the information abstract value is the same as those of the files exists in the cloud storage system to indicate that the data of the file has existed in the cloud storage system, the file is not uploaded any more, so that duplicate files are prevented from existing in the cloud storage system, and deduplication for the files of the cloud storage system is realized.

The invention provides a data transmission method and device and a readable storage medium. In the data transmission method, on one hand, a proxy server performs detection of corresponding interface functions on request data of a service system, performs duplicate removal processing on the request data of an interface with the same function, and then transmits the request data; or the request data of the interface of the function needing to be excluded is not transmitted, so that the interface desensitization of the service system is realized; and on the other hand, sensitive information detection is performed on the response data of the service system, and the response data containing sensitive information is transmitted after being desensitized, so that information desensitization of the service system is realized. Therefore, the data transmission method can realize function desensitization of the service system and improve the security of data transmission in the service system.

The embodiment of the invention discloses a road entity data deduplication method and device, computing equipment and a medium. The method comprises: acquiring road source data, classifying the road source data into at least one data subset according to road entity event types, one data subset corresponding to one road entity event type, and the road source data being used for describing the roadentity events; determining a road name and a geographic area name in the text content corresponding to each piece of road source data in each data subset; and according to the road name and the geographic area name in the text content corresponding to each piece of road source data, performing text matching in historical road source data belonging to the same road entity event type as the corresponding data subset, and determining a duplicate text in each data subset. According to the embodiment of the invention, the de-duplication effectiveness of the road entity data in the internet data canbe improved, so that the processing efficiency of mass road entity data is improved.

The invention discloses an SQL interaction data preprocessing method and device based on a transaction time sequence directed graph, and the method comprises the steps: carrying out normal form training according to original SQL interaction instruction data in a business system, obtaining a trained SQL execution instruction set SqlCmdSet, and constructing a trained business-oriented transaction time sequence directed graph; obtaining actual SQL interaction instruction data in a business system production environment, performing paradigm to obtain an actual SQL execution instruction set SqlCmdSet, and obtaining an actual transaction sequence set session Transaction Set; and matching the actual transaction sequence set session Transaction Set with a trained business-oriented transaction timesequence directed graph according to the actual transaction sequence set session Transaction Set, and outputting a graph path corresponding to the actual SQL interaction instruction data. Effective data can be reserved to the maximum extent, and useless data can be removed. Repeated parts of the same transaction operation in the original data are extracted to the maximum extent, and duplicate removal of the original data is achieved.

A garlic sowing machine comprises a rack, the rack is successively provided with a rotary tillage device, a seed fetching device and a compaction driving device from front to rear, a plurality of boat-shaped soil digging tools are arranged under the seed fetching device, and the compaction driving device is connected with the seed fetching device through sprocket and chain transmission. The garlic sowing machine has the advantages of high mechanization degree, good growth property after garlic is planted, high sowing efficiency and the like.

The invention provides an information processing method and device in a rating system, a computer readable storage medium and a program product, the method and device are applied to information processing nodes, the rating system comprises at least two information processing nodes, and the at least two information processing nodes share a target database. The method comprises the steps of obtaining at least one piece of to-be-processed media information; performing duplicate removal on the media information to be processed based on the shared target database to obtain a processing result; when the processing result represents that the to-be-processed media information is not stored in the target database, determining a rating label of the to-be-processed media information; and storing the to-be-processed media information carrying the rating label, and rating a target entity associated with the to-be-processed media information based on the rating label. Media information processing efficiency of the system is improved, and the exaggerated rating effect caused by data recording errors due to repeated statistics of the media information is reduced.

The invention discloses a data deduplication method and device for traffic playback, a test method and device, computer equipment and a storage medium, and the data deduplication method comprises the steps of collecting a service request; obtaining byte code information of the service request, and extracting features of the byte code information; generating a feature value corresponding to the feature of the byte code information, wherein the feature value is used for uniquely representing the byte code information; and de-duplicating the service request by using the feature value to obtain de-duplicated service request data. According to the invention, the byte code information of the service request can reflect the relevant characteristics of the service request from the code level and directly reflect the difference of different service requests, so that the relevant characteristics of the byte code information are utilized to perform de-duplication on the service request, thereby achieving the accurate de-duplication on the service flow.

The embodiment of the invention discloses a method and device for determining recalled interest point information, equipment and a medium, and relates to the field of retrieval. The method comprises the following steps: selecting a current main word from words included in a query sentence, and taking interest point information including the current main word as candidate interest point information; And selecting recall interest point information from the candidate interest point information according to the historical main word state of each word in the query sentence and the occurrence condition in the candidate interest point information. The embodiment of the invention provides a determination method and device for recall interest point information, equipment and a medium. The determined recall interest point information is different.

The invention discloses a data processing method, an intelligent terminal and a storage medium. The method comprises the following steps: receiving heartbeat data sent by a client application; importing the heartbeat data into a big data cluster; storing the heartbeat data in a bit encoding mode; and according to a used data warehouse tool, developing a user-defined bit code calculation function,and calling the user-defined function to perform bit code related calculation on the heartbeat data according to actual data analysis requirements. According to the method, the product use state of the user in a certain time period of the current day is recorded in a bit coding mode, the product use time distribution of the user in one day can be recorded in one field, the use condition of the user in the current day is recorded in a single field, and the use condition of the user in the current year can be stored in several fields in the current day; only a plurality of pieces of data of users need to be stored in one year, the storage space is greatly reduced, deduplication of heartbeat data in the same time period is achieved, related calculation is conducted on data based on bit coding, and the calculation efficiency can be improved.

The invention provides an artificial intelligence based voice searching method and device. The artificial intelligence based voice searching method comprises the steps of firstly receiving a first query sentence input by a user; then carrying out lexical analysis on the first query sentence and determining a first entity list included in the first query sentence; then carrying out pinyin labeling on each entity in the first entity list and determining a first pinyin list; then determining a second entity list in fuzzy matching with the first pinyin list based on a preset pinyin entity library and further determining a third entity list corresponding to voice input by the user; and then getting rid of one word fragment in a second query sentence which is composed of entities in the third entity list to obtain a third query sentence, wherein the word fragment has the pinyin edition distance to other word segments less than a first preset value. Therefore, influence from a voice identification error to search in a voice searching process is reduced, duplication elimination of repeated fragments is realized, the accuracy of voice searching is increased, and the user experience is improved.