31results about How to "Improve the efficiency of duplicate checking" patented technology

The invention discloses a method and system for eliminating duplication during data count as well as a server and a storage medium, applicable to data duplication elimination in big data. The method provided by the invention comprises the following steps: receiving a call request, and performing load balancing by utilizing a dubbo component; analyzing the request, and according to a preset duplication elimination rank parameter in the request, creating a corresponding quantity of redis data storage bitmaps on the server; and acquiring a duplication elimination content parameter and the duplication elimination rank parameter in the request, calculating by virtue of a Bloom Filter algorithm to obtain a duplication elimination result, when the duplication elimination rank is higher than grade1 and a duplication elimination result return value is 0, calculating one group of hash functions again, and then performing duplication elimination again by virtue of the Bloom Filter algorithm. Inthe method disclosed by the invention, the load balancing is performed by virtue of the dubbo component, and according to the preset duplication elimination grade, count duplication elimination at a corresponding grade is performed by virtue of the Bloom Filter algorithm, so that data can be efficiently and rapidly processed, the probability that the data is eliminated mistakenly can be greatly reduced, and duplication elimination accuracy is improved.

The embodiment of the invention discloses a video duplicate checking method and device, and the method comprises the steps: constructing a multi-modal feature vector of a to-be-processed video; carrying out neighbor retrieval in a video library based on the multi-modal feature vector, screening out candidate videos similar to the to-be-processed video, and obtianing a candidate video set; calculating the similarity between each candidate video and the to-be-processed video to obtain a similarity result; and determining whether the to-be-processed video passes duplicate checking detection or not according to the similarity result. According to the scheme, the video duplicate checking efficiency can be improved while the video duplicate checking accuracy is ensured.

The invention relates to a medical data duplicate checking and associating method and system. The method comprises the following steps of: (1) extracting core data items in to-be-processed medical data; (2) classifying the core data items; (3) respectively carrying out preliminary screening on the data items in an exclusion array and a fuzzy array; (4) carrying out deep screening on the data items in the core data items; (5) setting a threshold value M2 of a suspected duplicated data similarity and/or a threshold value M3 of suspected associated data; and (6) after artificially checking and judging the suspected duplicated and/or associated data, inputting the data which is judged as non-duplicated data into a medical database, and endowing the data which is judged to be associated with one or more corresponding association labels. Compared with the prior art, the method and system provided by the invention have the characteristics of being low in missed judging rate, low in wrong judging rate and high in duplicate checking efficiency, and do not have high requirement for the profession degree of artificial checking, so that the operation costs of the duplicate checking and associating are remarkably reduced.

A method for checking duplication of text is disclosed, A fingerprint sequence of duplicate text to be checked can be stored in a text fingerprint database in advance, After the target text is obtained, the target fingerprint sequence is generated, and then the similar fingerprint sequence of each fingerprint in the target fingerprint sequence is calculated to obtain the similar fingerprint sequence. Finally, the fingerprint sequence including the target fingerprint sequence or the similar fingerprint sequence in the text fingerprint database is determined, and obviously, the text corresponding to the fingerprint sequence is the text similar to the target text. It can be seen that the method can generate similar fingerprint sequences of target fingerprint sequences, When judging whether the duplicated text and the target text are similar, only the fingerprint sequence of the duplicated text to be checked can be judged whether the fingerprint sequence of the duplicated text includes thetarget fingerprint sequence or the similar fingerprint sequence, and the similarity calculation of the duplicated text and the target text is not needed, thus saving the calculation amount and improving the duplication checking efficiency of the text. In addition, the present application also provides a text duplication checking apparatus, an apparatus, and a computer-readable storage medium, thefunctions of which correspond to the functions of the above-described method.

The embodiment of the invention discloses a thesis duplicate checking method and device, equipment and storage media. The thesis duplicate checking method comprises the following steps that: in a duplicate checking display interface, providing at least two optional duplicate checking platforms and the introduction information of each optional duplicate checking platform for a user to select as least one duplicate checking platform from the at least two optional duplicate checking platforms as a standby duplicate checking platform according to requirements, wherein the introduction information contains a charging standard, and the standby duplicate checking platform shares the thesis provided by the user; and according to the thesis and the charging standard of each standby duplicate checking platform, determining a payment amount, and finishing payment in one time to enable each standby duplicate checking platform to start a duplicate checking operation. By use of the embodiment of the invention, at least two optional duplicate checking platforms are provided for users to carry out selection, the standby duplicate checking platform shares the thesis provided by the user, payment is finished in one time during payment, so that the user does not need to submit the thesis on a plurality of duplicate checking platforms, multiple payment is carried out so as to save thesis duplicate checking time, and duplicate checking efficiency is improved.

The invention discloses an electronic invoice duplicate checking method and system, and belongs to the technical field of electronic invoices, and the method comprises the steps: obtaining the title information of a to-be-detected electronic invoice; Analyzing the head-up information of the electronic invoice to be detected to obtain at least one of an invoice code, an invoice number, an amount, an invoice issuing date and a check code; Taking the analyzed information as a unique identifier, firstly comparing the electronic invoice title information with all invoices in an invoice input page,and then comparing the analyzed unique identifier with a unique identifier in a data table pre-stored in a database; And when any comparison result is the same, determining that the repeated electronic invoices exist. According to the method, whether the electronic invoice is reimbursed repeatedly or not is automatically verified before the electronic invoice is input, so that the duplicate checking efficiency of the electronic invoice is greatly improved.

The invention provides a duplicate checking method. The method comprises the following steps of the first step, using a Word2Vec model to train to obtain a sentence vector of an original sentence anda contrast sentence, wherein the sentence vector is obtained by integrating a word vector and a word characteristic vector; the second step, based on the sentence vector of the original sentence and the sentence vector of the contrast sentence, calculating to obtain the included angle between the sentence vector of the original sentence and the sentence vector of the contrast sentence; the third step, determining the similarity between the original sentence and the contrast sentence, wherein when the included angle is less than or equal to a threshold, it is determined that the original sentence is similar to the contrast sentence; when the angle is greater than the threshold, it is determined that the original sentence is not similar to the contrast sentence. The method comprehensively considers the word vector and the word characteristic vector, compared with the calculation time based on sentence coding, the calculation time of the method is obviously shortened, the introduction ofthe word characteristic vector has a certain solution effect on the situation that checking is difficult after synonym replacing, the problem that word changing, order changing and adding or deletionof words are difficult to check based on complete sentence comparison is solved, and on the whole, the method not only improves the accuracy of duplicate checking, but also improves the efficiency ofthe duplicate checking.

An embodiment of the invention provides an object uploading method. The method of a network side comprises the steps of receiving a duplicate checking request sent by a user terminal, wherein the duplicate checking request carries a data size of a specified object and a specified container identifier; searching for a first class object with the same data size as the specified object in a storage container corresponding to the specified container identifier; and if the first class object is not found, returning a first duplicate checking response to the user terminal, wherein the first duplicate checking response is used for instructing the user terminal to upload the specified object. According to the method, the duplicate checking efficiency and the object uploading efficiency of the user terminal can be improved. The invention furthermore provides an electronic device of the network side, an object uploading method of a user terminal side and the user terminal.

The invention provides a picture duplicate checking method and device and a computer readable storage medium, and the method comprises the steps: generating a corresponding character string for a picture to be subjected to duplicate checking based on the attribute information of the picture to be subjected to duplicate checking; performing picture duplicate checking calculation on the character string through a Bloom filter; and outputting a duplicate checking result through the Bloom filter. According to the technical scheme, the picture duplicate checking process is simplified, and the duplicate checking efficiency is improved.

The invention discloses a project duplicate checking method and system based on concurrent tasks. The method comprises four steps of carrying out the dynamic analysis on the Internet hot words and common words through the Internet technology, and forming a cloud lexicon; and matching the text information in the declaration material with a cloud word bank through a text matching method, segmentingthe declaration material into the semantic word segmentation factors, obtaining an optimal word segmentation scheme through weighted calculation, counting the word frequency and eliminating the high-frequency single words; and returning the similarity values of the current duplicate checking projects and the historical projects to the segmented word subset of the current duplicate checking projectand the segmented word subset of the historical project through a cosine similarity algorithm CosineSimilar. During the big data calculation, a high-capacity high-speed memory is utilized, the memorymanagement is reasonably used, the frequent read-write access of a hard disk is reduced, the concurrent multithreading tasks are started, the system resources are fully utilized, and the maximum frequency of a CPU is brought into play, so that the duplicate checking efficiency is improved.

The embodiment of the invention provides an article duplicate checking method and device, electronic equipment and a storage medium, and belongs to the technical field of artificial intelligence. The method comprises the following steps: inputting a feature vector of an article to be detected into a pre-trained duplicate checking model to obtain a related article related to the article to be detected, and determining a duplicate checking rate of the article to be detected by taking the related article as a reference. Wherein the duplicate checking model is obtained by performing joint training according to training data of a plurality of mutually independent article databases, so that the articles of the duplicate checking model are more comprehensive, the duplicate checking range is expanded, the majority of researchers, college students and teachers do not need to switch different duplicate checking platforms to perform duplicate checking on the articles, the duplicate checking efficiency is improved, and in addition, the duplicate checking efficiency is improved. The related articles of the to-be-queried article can be quickly screened out through the duplicate checking model, the subsequent repetition rate detection range of the to-be-detected article is narrowed, and the article duplicate checking accuracy can be improved.

The invention discloses an oil well indicator diagram data duplicate checking method which comprises the following steps: acquiring indicator diagram data of all oil wells in a well area within one day, and defining effective indicator diagram data; grouping and screening the indicator diagram data of all oil wells in one day, and counting the quantity of grouped effective indicator diagram data to obtain a repeated indicator diagram data sample and a repeated indicator diagram data test sample set; screening to-be-checked duplicate indicator diagram data of all oil wells according to the duplicate indicator diagram data inspection sample set to obtain all duplicate indicator diagram data and a duplicate indicator diagram data result set; according to the repeated indicator diagram data in the repeated indicator diagram data result set and the well names thereof, obtaining a repeated indicator diagram statistical result set based on the single well names; obtaining a well name association relation table of the repeated indicator diagram data according to the repeated indicator diagram data in the repeated indicator diagram data result set and different well repetition conditions thereof; and according to the well name association relation table of the repeated indicator diagram data, obtaining all different well repeated records of each piece of repeated indicator diagram data.

The invention discloses an intelligent sales performance assessment method based on a CRM system, and the method comprises the steps: collecting a plurality of sales behavior data in a plurality of time periods, and the parameters of the behavior data comprising the unit price x of a product, the number s of products, the clue y of the product, and the clue conversion rate b of the product; inputting the plurality of behavior data into a CRM system to obtain final sales performance A in a plurality of time periods, the CRM system comprising: a normalization module for normalizing parameter values of the plurality of behavior data to obtain normalized values of the parameter values of the plurality of behavior data; and a performance calculation module used for calculating the normalized values of the parameter values of the behavior data through a specific gravity formula to obtain a final sales performance A.

The invention discloses a data duplicate checking method and a data duplicate checking device. The data duplicate checking method is applied to a system comprising a source database, a client, a cacheand a result storage database. The method comprises: at a first moment, obtaining a duplicate checking request for data to be subjected to duplicate checking, the data to be subjected to duplicate checking having a unique identifier; for the to-be-duplicated data, judging whether a corresponding unique identifier exists in the result storage database or not; when judging that the unique identifier exists, obtaining a duplicate checking moment corresponding to the unique identifier; obtaining change data in a source database between a duplicate checking moment and a first moment, and performing duplicate checking comparison with the data to be subjected to duplicate checking; and storing a duplicate checking comparison result into the result storage database. The data duplicate checking efficiency can be improved, high duplicate checking accuracy can be guaranteed, and the method and the device can be suitable for first duplicate checking and subsequent multiple duplicate checking at the same time, that is, the efficiency of multiple duplicate checking in the first duplicate checking and business process can be improved.

The invention discloses an enterprise name duplicate checking method and device. The method comprises the following steps of: searching a second enterprise name matched with a first enterprise name tobe subjected to duplicate checking by utilizing ES; performing word segmentation on the first enterprise name and the second enterprise name according to structural elements, wherein the structural elements comprise administrative regions, company description and organization forms, and the company description comprises company word sizes and industry description; comparing each structural element in the first enterprise name with each structural element in the second enterprise name, and determining a first similarity corresponding to the administrative region, a second similarity corresponding to the company description and a third similarity corresponding to the organization form; determining the total similarity between each second enterprise name and the first enterprise name based on the first similarity, the second similarity and the third similarity; and determining the second enterprise name corresponding to the total similarity meeting the preset condition as the enterprisename which is the same as the first enterprise name. According to the invention, the duplicate checking precision and the duplicate checking efficiency can be improved.

The invention relates to the technical field of digital medical treatment. The invention discloses a structured medical record duplicate checking method which comprises the following steps: acquiring a structured medical record, filtering the structured medical record to obtain medical record data, and extracting one or more keywords in the medical record data; extracting 64-bit fingerprint features of the keywords, and performing weighted accumulation on the 64-bit fingerprint features corresponding to the keywords to obtain a 64-bit feature sequence string of the structured medical record; dividing the 64-bit feature sequence string into continuous 4 sections of 16-bit sub-sequence strings, generating a query statement according to the 4 sections of 16-bit sub-sequence strings of the structured medical record, the medical record category and the disease diagnosis code, and obtaining a query result from a medical record database based on the query statement; and determining the Hamming distance between the 64-bit feature sequence string of the structured medical record and the 64-bit feature sequence string of the medical record contained in the query result, and determining whether repeated medical records are queried according to the Hamming distance. According to the method, similar structured medical records can be quickly positioned, and the duplicate checking efficiency is higher.

The embodiment of the invention provides a data processing method and device, computer equipment and a computer readable storage medium, and the method comprises the steps: determining whether a cloud end has left-over data or not if the left-over data exists in a pre-sending area during data reporting, wherein the left-over data comprises data which is not deleted after the pre-sending area reports the data to the cloud end at the previous time; and if the left-over data does not exist in the cloud end, sending the to-be-reported data to the pre-sending area to instruct the pre-sending area to report the left-over data and the to-be-reported data to the cloud end. According to the embodiment of the invention, whether the left-over data exists in the cloud end or not can be determined when the left-over data exists in the pre-sending area, so that duplicate checking of the left-over data is realized, the duplicate checking efficiency can be improved, and the performance loss of computer equipment can be reduced; when the left-over data does not exist in the cloud end, the left-over data and the to-be-reported data can be reported to the cloud end together from the pre-sending area, so that the data reporting efficiency is improved.

The invention discloses a video duplicate checking method and device, a storage medium and a computer program product. The method comprises the following steps: acquiring N to-be-processed video frames and L reference video frames; the N to-be-processed video frames are determined from target videos needing duplicate checking, and the L reference video frames are determined from H reference videos used for duplicate checking comparison; based on the image feature similarity between each to-be-processed video frame and each reference video frame, determining M reference video frames from the L reference video frames as similar video frames; one similar video frame is similar to at least one video frame to be processed; determining K reference videos from the H reference videos as similar videos according to the M similar video frames; any similar video comprises at least one similar video frame; determining the audio feature similarity between the target video and each similar video, and judging whether the target video is repeated with each similar video according to the audio feature similarity; the video duplicate checking efficiency can be improved.