Anomaly detection for web document revisions

Abstract

The present application relates to anomaly detection for web document revisions, including an anomaly detection method for web document revisions, comprising: accessing a published web document, the published web document including one or more different elements generated by a computing device associated with a user; accessing an updated web document, the updated web document being generated based on one or more modifications to the published web document; generating one or more anomaly scores based on a comparison of the updated web document with the published web document, the one or more anomaly scores providing a measure of a deviation of the updated web document from the published web document; and blocking publication of the updated web document based on the one or more anomaly scores generated based on the comparison of the updated web document with the published web document, wherein blocking publication of the updated web document causes the updated web document to be invisible on one or more computing devices.

Description

[0001] This application is a divisional application of the PCT international application PCT / US2017 / 038593 filed on June 21, 2017, which entered the Chinese national phase with application number 201780038502.5 and invention name “Anomaly Detection of Web Document Revisions”.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This international application claims priority to U.S. patent application serial number 15 / 188,532, filed on June 21, 2016, entitled “ANOMALY DETECTION FOR WEBDOCUMENT REVISION,” the entire contents of which are incorporated herein by reference in their entirety. Technical Field

[0004] The present disclosure generally relates to the technical field of special-purpose machines configured to facilitate digital content management, including computerized variations of such special-purpose machines and improvements thereto, and to techniques that improve such special-purpose machines over other special-purpose machines for facilitating digital content management. In particular, the present disclosure provides systems and methods for detecting anomalous revisions of published digital content. Background Art

[0005] Many online content publishing platforms allow users to generate and publish content in the form of web documents (e.g., web pages) online, which can be browsed by other users using web browsers or applications. Typically, a uniform resource identifier (URI) is assigned to each published web document at or before the time of publication. Typically, these online content publishing platforms allow users to revise content even after publishing the content. Although some content of a web document can be updated, the URI typically remains unchanged. In some cases, allowing a web document to be revised while maintaining the same URI may be problematic for an online content publishing platform.

[0006] In one example, a content publishing platform is an online marketplace that allows users to create content in the form of product listings to offer their products for sale to other users. A user of the online marketplace may initially create a product listing for a first product that is in high demand, and the online marketplace may then assign a URI to the product listing. The user may then completely revise the product listing to cover a second product that has lower demand and desire, and the URI assigned to the product listing will remain the same, although the URI may still be associated with the inventory and historical sales of the first product that is in high demand. In this way, a user may exploit the online marketplace's revision capabilities to manipulate the search ranking of undesirable products, hide lower demand, manipulate historical sales figures, or otherwise manipulate consumer demand for products. While this may prove beneficial to individual users, it results in an overall decline in the online marketplace's navigation quality, information accuracy, and overall performance and reputation. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The various accompanying drawings depict only example embodiments of the present disclosure and are not to be considered as limiting its scope.

[0008] Figure 1 is a network diagram illustrating a content publishing platform having a client-server architecture configured for exchanging data over a network according to an example embodiment.

[0009] Figure 2 is a block diagram illustrating various functional components of an anomaly detection system provided as part of a content publishing platform according to an example embodiment.

[0010] Figure 3 is an interaction diagram illustrating an example exchange between a publishing system and a content publishing platform according to an example embodiment.

[0011] Figure 4A is an interface diagram illustrating a published web document according to an example embodiment.

[0012] Figure 4B is an interface diagram illustrating an updated version of a published web document according to an example embodiment.

[0013] Figures 5 to 8 is a flowchart illustrating an example operation of an anomaly detection system according to an example embodiment when providing anomaly detection services for a content publishing platform.

[0014] Figure 9 is a flowchart illustrating an example operation of a content publishing system according to an example embodiment when providing a user-generated content publishing service.

[0015] Figure 10is a diagrammatic representation of a machine in the example form of a computer system within which a set of instructions may be executed to cause the machine to perform any one or more of the methodologies discussed herein. DETAILED DESCRIPTION

[0016] Reference will now be made specifically to specific exemplary embodiments for carrying out the subject matter of the present invention. Examples of these specific embodiments are shown in the accompanying drawings. It will be understood that these examples are not intended to limit the scope of the claims to the illustrated embodiments. Rather, they are intended to encompass alternatives, modifications, and equivalents that may be included within the scope of this disclosure. In the following description, specific details are set forth to provide a thorough understanding of this subject matter. The embodiments may be practiced without some or all of these specific details.

[0017] Aspects of the present disclosure relate to systems and methods for detecting anomalies in web document revisions. As used herein, an "anomaly" includes a modification to a web document that alters the web document beyond an expected threshold of changes. In an example embodiment, the web document is a web page, such as an online marketplace listing for a product. The disclosed subject matter is applicable to any online content publishing platform that allows users to revise published content. In one example, the online content publishing platform is an online marketplace.

[0018] Example embodiments relate to web documents that include a plurality of different elements, such as images, text, and numerical values. In the example web document of a list item, the different elements may include an image of a product, a text description of the product, and a price. In these example embodiments, a method may include accessing an updated version (e.g., a user-revised version) of a published web document. The method also includes comparing various portions of the published (e.g., unchanged) web document with corresponding portions of the updated document to generate various anomaly scores. For example, a published image included in a published web document may be compared with an updated image included in an updated web document to generate an image similarity score; published text included in a published web document may be compared with updated text included in an updated web document to generate a text match score; and an updated numerical value included in an updated web document may be compared with a published numerical value (or an average of previously published associated numerical values) to generate a numerical deviation score.

[0019] The method may also include determining whether to publish the updated web document based on a comparison of the anomaly score to a threshold anomaly score. In some embodiments, the system may prevent publishing the updated web document if any one of the anomaly scores is above the threshold score. In other embodiments, the system may prevent publishing the updated web document if a combination of the anomaly scores is above the threshold score. Other aspects of the present disclosure include using machine learning techniques to revise the threshold score based on manual review of revision list items in which anomalies were detected.

[0020] refer to Figure 1 , shows an example embodiment of a high-level client-server based architecture 100. Although Figure 1 A client-server based architecture 100 is shown, but the subject matter of the present invention is of course not limited to such an architecture and is equally well suited for example to event-driven, distributed or peer-to-peer architecture systems. Furthermore, in order to avoid obscuring the subject matter of the present invention with unnecessary detail, Figure 1 Various functional components that are not relevant to conveying an understanding of the subject matter of the present invention are omitted. Figure 1 The various functional components shown in FIG. 5 are discussed in the singular, but multiple instances of any of the various functional components may be employed.

[0021] A content publishing platform 102, in the exemplary form of a network-based marketplace, provides server-side functionality to one or more client devices 110 via a network 104 (e.g., the Internet or a wide area network (WAN)). Figure 1 Shown are a web client 112 (e.g., a browser), a client application 114, and a programmatic client 116 executing on the client device 110. One or more portions of the network 104 may be an adhoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), a portion of the Internet, a portion of the public switched telephone network (PSTN), a cellular telephone network, a wireless network, a WiFi network, a WiMax network, another type of network, or a combination of two or more such networks.

[0022] The client device 110 may include, but is not limited to, a mobile phone, a desktop computer, a laptop computer, a portable digital assistant (PDA), a smart phone, a tablet computer, an ultrabook, a netbook, a notebook computer, a multiprocessor system, a microprocessor-based or programmable consumer electronic product, a game console, a set-top box, or any other communication device that a user can use to access the content publishing platform 102. In some embodiments, the client device 110 may include a display module (not shown) to display information (e.g., in the form of a user interface). In other embodiments, the client device 110 may include one or more of a touch screen, an accelerometer, a gyroscope, a camera, a microphone, a global positioning system (GPS) device, etc. In one embodiment, the content publishing platform 102 is a network-based marketplace that publishes announcements (e.g., web documents) including listings of products available on the network-based marketplace.

[0023] One or more users 106 may be humans, machines, or other devices that interact with client devices 110. In an example embodiment, users 106 are not part of the client-server based architecture 100, but may interact with the client-server based architecture 100 via client device 110 or another device. For example, user 106 may provide input (e.g., touch screen input or alphanumeric input) to client device 110, and transmit the input to content publishing platform 102 via network 104. In this example, content publishing platform 102 transmits information to client device 110 via network 104 in response to the input received from user 106 for presentation to user 106. In this manner, user 106 may interact with content publishing platform 102 using client device 110.

[0024] The client device 110 may include one or more client applications 114 (also referred to as "apps"), such as, but not limited to, a web browser, a messaging application, an email application, an e-commerce site application (also referred to as a marketplace application), and the like. In some embodiments, if an e-commerce site application is included in the client device 110, the application may be configured to locally provide a user interface and at least some of the functionality, wherein the client application 114 is configured to communicate with the content publishing platform 102 as needed to obtain data or processing capabilities that are not available locally (e.g., access a database of items available for sale, authenticate the user 106, verify a payment method, etc.). Conversely, if an e-commerce site application is not included in the client device 110, the client device 110 may use its web browser to access an e-commerce site (or a variant thereof) hosted on the content publishing platform 102.

[0025] An application program interface (API) server 120 and a network server 122 are coupled to an application server 140 and provide a programming interface and a web interface, respectively, to the application server 140. The application server 140 can host a publishing system 142 and an anomaly detection system 144, each of which can include one or more modules or applications 114 and can be embodied as hardware, software, firmware, or any combination thereof. Next, the application server 140 is shown coupled to a database server 124 that facilitates access to a database 126. In an example embodiment, the database 126 is a storage device that stores information (e.g., publications or listings) to be posted to the publishing system 142. According to an example embodiment, the database 126 can also store digital item information.

[0026] Additionally, a third-party application 132 executing on a third-party server 130 is shown having programmatic access to the content publishing platform 102 via a programming interface provided by the API server 120. For example, the third-party application 132 utilizes information retrieved from the content publishing platform 102 to support one or more features or functions on a website hosted by the third party.

[0027] The publishing system 142 provides a number of publishing functions and services to users 106 accessing the content publishing platform 102. For example, the publishing system 142 provides an interface that allows users 106 to create and publish web documents using a client device 110 (e.g., by communicating with the client device 110). The publishing system 142 may also provide an interface that allows users 106 to modify various portions of a published web document.

[0028] The anomaly detection system 144 is configured to monitor changes made by users 106 to published web documents in order to detect anomalous updates. To do so, the anomaly detection system 144 compares updated web documents with previous or original versions of the web documents to generate various anomaly scores based on the extent to which the web documents have been modified. The anomaly detection system 144 can flag certain updated web documents based on these anomaly scores to prevent the publishing system 142 from publishing the updated web documents.

[0029] Although the publishing system 142 and the anomaly detection system 144 Figure 1 102 (e.g., publishing system 142 and anomaly detection system 144 are subsystems of content publishing platform 102), but it should be understood that in alternative embodiments, each system 142 and 144 may form part of a separate and distinct service or platform from content publishing platform 102. In some embodiments, anomaly detection system 144 may form part of publishing system 142.

[0030] Figure 2 1 is a block diagram illustrating various functional components of anomaly detection system 144 according to an example embodiment. To avoid obscuring the subject matter of the present invention with unnecessary detail, various functional components (e.g., modules, engines, and databases) that are not closely related to understanding the subject matter of the present invention have been described in detail. Figure 2 However, those skilled in the art will readily recognize that various additional functional components may be supported by the anomaly detection system 144 to facilitate additional functionality not specifically described herein.

[0031] As shown, the anomaly detection system 144 includes an anomaly detector 200, a decision module 240, and an improvement module 250. The anomaly detector 200 includes a text anomaly detector 210, an image anomaly detector 220, and a numerical anomaly detector 230. Each of the above-referenced functional components of the anomaly detection system 144 is configured to communicate with each other (e.g., via a bus, shared memory, switch, application programming interface (API)). Figure 2 Any one or more functional components shown in and described herein may be implemented using hardware (e.g., a processor of a machine) or a combination of hardware and software. For example, any module described herein may configure a processor to perform the operations described herein for that module. In addition, any two or more of these modules may be combined into a single module, and the functionality described herein for a single module may be subdivided into multiple modules. Furthermore, according to various example embodiments, Figure 2 Any functional components shown in the may be implemented together or separately within a single machine, database, or device, or may be distributed across multiple machines, databases, or devices.

[0032] The text anomaly detector 210 is responsible for detecting anomalies in a revised web document that occur due to modifications to text included in one or more previously published web document versions. To this end, the text anomaly detector 210 is configured to compare the modified text included in the revised web document with the published text included in the published web document. Using this comparison as a basis, the text anomaly detector 210 generates a text match score that provides a measure of similarity between the modified text and the published text. When determining whether the modified text represents an anomaly, the text anomaly detector 210 compares the text match score with a threshold text match score. If the text match score exceeds the threshold text match score, the text anomaly detector 210 determines that there is an anomaly with respect to the modified text.

[0033] Image anomaly detector 220 is responsible for detecting anomalies in a revised web document that occur due to modifications to images included in one or more previously published versions of the web document. To this end, image anomaly detector 220 is configured to compare the modified image included in the revised web document with the published image included in the published web document. Using this comparison as a basis, image anomaly detector 220 generates an image similarity score that provides a measure of similarity between the modified image and the published image. When determining whether the modified image represents an anomaly, image anomaly detector 220 compares the image similarity score to a threshold image similarity score. If the image similarity score exceeds the threshold image similarity score, image similarity detector 220 determines that an anomaly exists with respect to the modified image.

[0034] In some embodiments, image anomaly detector 220 extracts keypoints from the original image and then stores the keypoints in database 126. The updated image is compared to the original published image by comparing each feature from the new image with the features of the original image stored in database 126 and finding candidate matching features based on the Euclidean distance of their feature vectors. From the full set of matches, image anomaly detector 220 identifies a subset of keypoints in the new image that are consistent in terms of objects and their positions, scales, and orientations, in order to filter out good matches.

[0035] The numerical anomaly detector 230 is responsible for detecting anomalies in a revised web document that occur due to modifications to numerical values included in one or more previously published web document versions. To this end, the numerical anomaly detector 230 is configured to compare the modified numerical value included in the revised web document with one or more numerical values included in one or more published web documents. In some embodiments, the numerical anomaly detector 230 compares the modified numerical value with the published numerical value included in the published version of the same web document. In other embodiments, the numerical anomaly detector 230 compares the modified numerical value with the average of multiple numerical values included in the associated published web document. For example, a web document may include a product price, and the numerical anomaly detector 230 may compare the modified product price with an average price of the product calculated based on multiple listings of the product.

[0036] Using this comparison as a basis (e.g., a comparison with a single previously published value or an average of published values), the value anomaly detector 230 generates a value deviation score that indicates the deviation of the modified value from one or more published values. In determining whether the modified value represents an anomaly, the value anomaly detector 230 compares the value deviation score with a threshold value deviation score. If the value deviation score exceeds the threshold value deviation score, the value anomaly detector 230 determines that an anomaly exists with respect to the modified value.

[0037] Each of the text matching score, image similarity score, and numerical deviation score can be considered an anomaly score. The text anomaly detector 210, the image anomaly detector 220, and the numerical anomaly detector 230 provide the anomaly score and an indication (e.g., a marker) of whether an anomaly exists in each portion (e.g., text, image, or numerical value) of the modified web document to the decision module 240. The decision module 240 uses the information provided by the text anomaly detector 210, the image anomaly detector 220, and the numerical anomaly detector 230 to determine whether to allow the modified web document to be published. In some embodiments, if an anomaly exists in any portion (e.g., text, image, or numerical value) of the modified web document, the decision module 240 can prevent the modified web document from being published.

[0038] In some embodiments, the decision module 240 aggregates the individual anomaly scores (e.g., text matching score, image similarity score, and numerical deviation score) to generate an aggregate anomaly score. For example, the decision module 240 may sum each corresponding anomaly score to generate the aggregate anomaly score. The decision module 240 may further compare the aggregate anomaly score with a threshold aggregate anomaly score to determine whether to allow the updated web document to be published. If the aggregate anomaly score exceeds the threshold anomaly score, the decision module 204 blocks the publication of the modified web document.

[0039] The improvement module 250 is configured to improve various threshold scores (e.g., a threshold text match score, a threshold image similarity score, a threshold numerical deviation score, and a threshold aggregate anomaly score) based on the results of a manual review of the modified web document that is determined to be anomaly. For example, upon deciding to block the publication of a modified web document, the decision module 240 may mark the modified web document for manual review by an administrator user. Upon manual review of the modified web document, if the administrator user determines that no anomalies exist, the improvement module 250 may revise one or more threshold anomaly scores, for example, by raising the thresholds.

[0040] Figure 31 is an interaction diagram illustrating an example exchange between a publishing system 142 and anomaly detection system 144 according to an example embodiment. In operation 302, the publishing system 142 publishes a web document generated using data received from a client device 110 based on information provided by a user 106. Upon generation, the web document is assigned a URI that uniquely identifies the web document. The web document can be accessed and displayed on the client device 110 or other such device using a web client 112, a client application 114, or a programmatic client 116. For example, the client device 110 submits a request for a particular document to the application server 140, and the application server 140 responds to the client device 110 with the web document and any other data required by the client device 110 to display the web document.

[0041] For example, Figure 4A A web document 400 is shown that illustrates an exemplary form of a marketplace product listing. As shown, web document 400 includes a number of different elements, including: text 402—the title of the product; image 404—the image of the product; and value 406—the price of the product. Web document 400 also includes a URI 408 that uniquely identifies web document 400.

[0042] Back to Figure 3 In operation 304, the publishing system 142 provides a user interface for revising a published web document (e.g., published web document 400) to the client device 110. The user interface includes a plurality of fields (e.g., text 402, image 404, and value 406) corresponding to a plurality of different elements of the published web document, which allows the user 106 to modify each portion of the web document.

[0043] At operation 306, the publishing system 142 generates an updated web document based on the modifications to the published web document. As an example, Figure 4B Shown is an updated web document 450 generated based on the modifications to web document 400. As shown, although text 402 has changed to text 452, image 404 has changed to image 454, and value 406 has changed to value 456, updated web document 450 includes URI 408, which is the same as the URI in web document 400.

[0044] At operation 308, the anomaly detection system 144 accesses the updated web document (e.g., updated web document 450) generated by the publishing system 142. At operation 310, the anomaly detection system 144 analyzes the updated web document. When analyzing the updated web document, the anomaly detection system 144 compares various updated portions of the updated web document with various portions of the published web document (published in operation 302) to generate an anomaly score to be used as a basis for determining whether the updated web document is anomalous.

[0045] At operation 312, the anomaly detection system 144 determines whether the updated web document is anomaly. In other words, the anomaly detection system 144 determines whether the updated web document includes an anomaly due to one or more modifications made by the user 106 using the user interface provided at operation 304. The determination of whether the updated web document is anomaly is based on the anomaly detection system 144 determining whether one or more anomaly scores (e.g., a text matching score, an image similarity score, a numerical deviation score, and an aggregate anomaly score) crosses (e.g., is greater than) a corresponding threshold anomaly score.

[0046] If the anomaly detection system 144 determines that the updated web document is not anomalous in operation 312 , the anomaly detection system 144 allows publishing of the updated web document in operation 314 , and the publishing system 142 publishes the updated web document in operation 316 .

[0047] If the anomaly detection system 144 determines that the updated web document is anomaly at operation 312, the anomaly detection system 144 prevents publishing of the updated web document at operation 318. For example, the anomaly detection system 144 may instantiate a flag that, when read by the publishing system 142, causes the publishing system 142 to end the publishing process with respect to the updated web document.

[0048] In response to the anomaly detection system 144 preventing the updated web document from being published, the publishing system 142 generates a message in operation 320 to notify the user 106 that modifications to the web document will not be allowed and that the updated web document will not be published due to the detected anomaly. In operation 322, the publishing system 142 sends the message to the client device 110 of the user 106. The publishing system 142 can utilize any of a number of messaging networks and platforms to transmit the message to the client device 110 of the user 106. For example, the publishing system 142 can push a notification (e.g., via a related push notification service), email (e-mail), instant message (IM), short message service (SMS), text, fax, or voice (e.g., voice over IP (VoIP)) message via a wired (e.g., Internet), plain old telephone service (POTS), or wireless (e.g., mobile, cellular, WiFi, WiMAX) network.

[0049] Figure 5 is a flow chart illustrating example operations of anomaly detection system 144 according to an example embodiment when executing method 500 for providing anomaly detection services for content publishing platform 102. Method 500 may be implemented as computer-readable instructions that are executed by one or more processors, such that the operations of method 500 may be performed in part or in whole by anomaly detection system 144; therefore, method 500 is described below by way of example with reference thereto. However, it should be understood that at least some operations of method 500 may also be deployed on various other hardware configurations, and method 500 is not intended to be limited to anomaly detection system 144.

[0050] At operation 505, the anomaly detection system 144 accesses a published web document (e.g., published web document 400). The published web document includes a plurality of different elements, such as text, one or more images, and one or more numerical values. The different elements may be generated by or based on information from a user 106 of the content publishing platform 102. In the example, the published web document corresponds to a marketplace product listing and includes a text description of the product, one or more images of the product, and a price for the product.

[0051] At operation 510, the anomaly detection system 144 accesses an updated (modified) web document (e.g., updated web document 450). The updated web document is based on one or more modifications to the published web document (e.g., initiated by user 106). Like the published web document, the updated web document includes a plurality of different elements (e.g., text, one or more images, and one or more numerical values). The updated web document includes at least one user revision (e.g., modification) to a different element of the published web document, and in some cases may include at least one user revision to each portion of the published web document.

[0052] The updated web document may be or include a data object stored in the database 126 and may be generated based on user input (e.g., user revisions) received from a user interface that allows the user 106 to edit the published web document. In addition, the published web document and the updated web document are assigned the same URI 408.

[0053] At operation 515, the anomaly detector 200 generates one or more anomaly scores. Each of the one or more anomaly scores provides a measure of the deviation of the updated web document from the published web document. The one or more anomaly scores may include one or more of a text match score, an image similarity score, and a numerical deviation score. Thus, generating the one or more anomaly scores may include: calculating a text match score based on a comparison of the updated text (e.g., update text 452) included in the updated web document with the published text (e.g., published text 402) included in the published web document; calculating an image similarity score based on a comparison of the updated image (e.g., update image 454) included in the updated web document with the published image (e.g., published image 404) included in the published web page; and calculating a numerical deviation score based on a difference between an updated numerical value (e.g., update numerical value 456) included in the updated web document and an average value (e.g., product average price) associated with the web document.

[0054] In some embodiments, the one or more anomaly scores include an aggregate anomaly score. Thus, in these embodiments, generating the one or more anomaly scores may further include aggregating the text matching score, the image similarity score, and the numerical deviation score to generate the aggregate anomaly score.

[0055] At operation 520, the anomaly detection system 144 detects whether the updated web document includes an anomaly. Detecting an anomaly in the updated web document includes comparing one or more anomaly scores to one or more corresponding threshold anomaly scores. In some embodiments, detecting an anomaly includes any of the following: determining that a text match score crosses a threshold text match score; determining that an image similarity score crosses a threshold image similarity score; or determining that a numerical deviation score crosses a threshold numerical deviation score. In embodiments where the one or more anomaly scores include an aggregated anomaly score, detecting an anomaly in the updated web document may include determining that the aggregated anomaly score crosses a threshold anomaly score.

[0056] At operation 520, if the anomaly detection system 144 detects an anomaly in the updated web document, then at operation 525, the anomaly detection system 144 prevents the updated web document from being published. For example, the anomaly detection system 144 may instantiate a flag associated with the updated web document that causes the publishing system 142 to stop publishing the updated web document.

[0057] At operation 520, if anomaly detection system 144 does not detect an anomaly in the updated web document, then at operation 530, anomaly detection system 144 allows the updated web document to be published. According to the above example, anomaly detection system 144 does not instantiate a flag associated with the anomaly, and thus, publishing system 142 continues to publish the updated web document. In another example, anomaly detection system 144 may instantiate a different flag that signals publishing system 142 to continue publishing the updated web document.

[0058] like Figure 6 As shown, method 500 includes operations 605, 610, 615, 620, 625, and 630. In some example embodiments, operations 605, 610, 615, 620, 625, and 500 included in method 500 may be performed before or as part of operation 515 of method 500 (e.g., a previous task, subroutine, or portion), where anomaly detection system 144 generates one or more anomaly scores.

[0059] In operation 605, the text anomaly detector 210 compares the published text (e.g., published text 402) included in the published web document (e.g., published web document 400) with the updated text (e.g., updated text 452) included in the updated web document (e.g., updated web document 450). In operation 610, the text anomaly detector 210 generates a text matching score based on the comparison of the published text and the modified text. The text matching score provides a measure of similarity between the published text and the modified text.

[0060] For example, the text matching score can be or include a cosine similarity score, which provides a similarity measure between two vectors in an inner product space that measures the cosine angle between them. Thus, in some embodiments, generating the text matching score can include applying a cosine similarity algorithm to two vectors—a first vector corresponding to the posted text and a second vector corresponding to the updated text.

[0061] In the first example, the posting text might include "Headset work Mobile phones Driving Sunglass Headphone BTGlass Wireless Bluetooth," and the modified text might include "Headsetwork Mobile phones Driving Sunglass Headphone BTGlass Wireless Bluetooth-withextra protection." In this example, the posting text has been modified to include the additional attribute "with extraprotection." Applying the cosine similarity algorithm in this example returns a cosine similarity score of 87.71.

[0062] In the second example, the published text may include "Headset work Mobile phones Driving Sunglass Headphone BTGlass Wireless Bluetooth," and the modified text may include "Headset Driving Sunglass Wireless Bluetooth Headphone BTGlass work Mobile phones." In this example, the published text has been modified by rearranging the order of some words. In this example, applying the cosine similarity algorithm returns a cosine similarity score of 99.99.

[0063] In a third example, the published text may include "Headset work Mobile phones Driving Sunglass Headphone BTGlass Wireless Bluetooth," and the modified text may include "UniversalQi Wireless Charger Receiver Charging Pad iphone android htc sony." In this example, the published text has been completely changed. Applying the cosine similarity algorithm in this example returns a cosine similarity score of 9.53.

[0064] In operation 615, the image anomaly detector 220 compares the published image (e.g., published image 404) included in the published web document with the updated image (e.g., updated image 454) included in the updated web document. In operation 620, the image anomaly detector 220 generates an image similarity score based on the comparison of the published image and the updated image. The image similarity score provides a measure of similarity between the published image and the updated image. Figure 7 , discussing further details regarding operation 620 according to some example embodiments.

[0065] At operation 625, the value anomaly detector 230 generates a value deviation score based on an analysis of the updated (e.g., modified) value (e.g., the updated value 456) included in the updated web document. In some embodiments, the value anomaly detector 230 generates the value deviation score by calculating the difference between the published value and the updated value.

[0066] In other embodiments, the numerical anomaly detector 230 generates a numerical deviation score based on a comparison of the updated numerical value with an average of numerical values associated with the published web document. For example, a published web document may include a marketplace listing for a product for sale, and the published numerical value may be the price of the product. In this example, the numerical anomaly detector 230 may determine the average price of the product based on, for example, other listings for the product published by the content publishing platform 102. The numerical anomaly detector 230 calculates the difference between the updated price of the product and the calculated average price of the product.

[0067] In some embodiments, operation 630 is optional, where the decision module 240 aggregates the text matching score, the image similarity score, and the numerical deviation score to generate an aggregate anomaly score. For example, the decision module 240 may sum (weighted or unweighted) the text matching score, the image similarity score, and the numerical deviation score to generate an aggregate anomaly score.

[0068] like Figure 7 As shown, method 500 may include additional operations 705, 710, 715, 720, and 725. In some example embodiments, operations 705, 710, 715, 720, and 725 included in method 500 may be performed before or as part of (e.g., a pre-task, subroutine, or portion of) operation 620 of operation 515 of method 500, where image anomaly detector 220 generates an image similarity score.

[0069] At operation 705, the image anomaly detector 220 extracts a first set of feature descriptors from a published image (e.g., published image 404). The first set of feature descriptors includes key points of interest in the published image that provide a "feature description" of the published image. In order to perform reliable recognition, it is important to be able to detect the features extracted from the original image even in the presence of changes in image scale, noise, and lighting. These points are typically located in high contrast areas of the image, such as edges of objects. Similarly, if any changes in the internal geometry of the bent or flexible object occur between two images in the processed set, then features located in the bent or flexible object may be inappropriate. Therefore, the image anomaly detector 220 can extract a large number of features from the image to reduce the contribution of errors caused by these local changes to the average error of all feature matching errors.

[0070] In some embodiments, the image anomaly detector 220 may utilize a scale-invariant feature transform (SIFT) algorithm to extract image feature descriptors. Using SIFT, the image anomaly detector 220 transforms the published image into a large set of feature vectors, each of which is invariant to image translation, scaling, and rotation, particularly partially invariant to illumination changes, and robust to local geometric distortions. The key locations used for extraction may include the maximum and minimum values of the results of the difference between a Gaussian function applied in scale space and a series of smoothed and resampled images. The image anomaly detector 220 may discard low-contrast candidate points and edge-response points along edges. The image anomaly detector 220 assigns a primary orientation to a local key point. The image anomaly detector 220 may then obtain an image feature descriptor by blurring and resampling the local image orientation plane, taking into account pixels around the radius of the key location.

[0071] In operation 710, the image anomaly detector 220 stores a first set of feature descriptors in a first matrix corresponding to the published image. A matrix is an array data structure comprising a set of elements, where each element is identified by at least one index or key.

[0072] At operation 715, the image anomaly detector 220 extracts a second set of image feature descriptors from the update image (e.g., update image 454). Similar to the first set of image feature descriptors, the second set of image descriptors includes key points of interest in the update image that provide a "feature description" of the update image. The image anomaly detector 220 extracts the second set of image feature descriptors from the update image in a manner similar to that discussed above with respect to extracting the first set of image descriptors from the release image. At operation 720, the image anomaly detector 220 stores the second set of feature descriptors in a second matrix corresponding to the update image.

[0073] In operation 725, the image anomaly detector 220 compares the first matrix with the second matrix to determine the similarity between the published image and the updated image. If both images show the same object, the first matrix and the second matrix will have similarity, otherwise the two matrices will be dissimilar. In addition, if A corresponds to the first matrix and B corresponds to the second matrix, then A·B -1 Should be equal to the identity matrix.

[0074] Consistent with some embodiments, comparing the first matrix to the second matrix may include matching image feature descriptors between the first and second matrices. A best candidate match for each feature descriptor in the second matrix is found by identifying its nearest neighbors in the first matrix. A nearest neighbor is defined as the feature descriptor with the smallest Euclidean distance from a given descriptor vector. Image anomaly detector 220 may, for example, identify nearest neighbors using a best-bin-first algorithm, which is a variation of a kd-tree algorithm.

[0075] The image anomaly detector 220 generates an image matching score based on the comparison of the first matrix to the second matrix. In an example, the image matching score generated by the image anomaly detector 220 is equal to the number of image feature descriptors that match between the two matrices divided by the total number of image feature descriptors.

[0076] like Figure 8 As shown, method 500 may include additional operations 805, 810, 815, 820, and 825. In some example embodiments, operations 805, 810, 815, 820, and 825 included in method 500 may be performed before or as part of (e.g., a pre-task, subroutine, or portion of) operation 520 of method 500, where anomaly detection system 144 determines whether an anomaly is detected in the updated web document.

[0077] In operation 805, the text anomaly detector 210 compares the text match score with a threshold text match score to determine whether the text match score exceeds the threshold text match score. The threshold text match score can be a default value set by an administrator of the content publishing platform 102, or can be a value improved by the improvement module 250 using machine learning technology. The threshold text match score can be a minimum value or a maximum value. Therefore, a text match score can be considered to have exceeded the threshold text match score because the text match score is higher than the maximum text match score or lower than the minimum text match score.

[0078] In operation 805, if the text anomaly detector 210 determines that the text matching score exceeds the threshold text matching score, then in operation 825, the text anomaly detector 210 determines that an anomaly exists in the updated web document with respect to the updated text, and the decision module 240 determines that an anomaly exists in the updated web document. In operation 805, if the text anomaly detector 210 determines that the text matching score does not exceed the threshold text matching score, the method continues to operation 810.

[0079] In operation 810, the image anomaly detector 220 compares the image similarity score to a threshold image similarity score to determine whether the image similarity score exceeds the threshold image similarity score. The threshold image similarity score may be a default value set by an administrator of the content publishing platform 102, or may be a value improved by the improvement module 250 using machine learning techniques. The threshold image similarity score may be a minimum value or a maximum value. Thus, an image similarity score may be considered to have exceeded the threshold image similarity score because the image similarity score is higher than the maximum image similarity score or lower than the minimum image similarity score.

[0080] In operation 810, if the image anomaly detector 220 determines that the image similarity score exceeds the threshold image similarity score, then in operation 825, the image anomaly detector 220 determines that an anomaly exists in the updated web document relative to the updated image, and the decision module 240 determines that an anomaly exists in the updated web document. In operation 810, if the image anomaly detector 220 determines that the image similarity score does not exceed the threshold image similarity score, the method continues to operation 815.

[0081] At operation 815, the numerical anomaly detector 230 compares the numerical deviation score with a threshold numerical deviation score to determine whether the numerical deviation score crosses the threshold numerical deviation score. The threshold numerical deviation score can be a default value set by an administrator of the content publishing platform 102, or can be a value improved by the improvement module 250 using machine learning techniques. The threshold numerical deviation score can be a minimum value or a maximum value. Therefore, a numerical deviation score can be considered to have crossed the threshold numerical deviation score because the numerical deviation score is higher than the maximum numerical deviation score or lower than the minimum numerical deviation score.

[0082] At operation 815, if the numerical anomaly detector 230 determines that the numerical deviation score crosses the threshold numerical deviation score, then at operation 825, the numerical anomaly detector 230 determines that an anomaly exists in the updated web document relative to the updated numerical value, and the decision module 240 determines that an anomaly exists in the updated web document. At operation 815, if the numerical anomaly detector 230 determines that the numerical deviation score does not cross the threshold numerical deviation score, the method continues to operation 820, where the decision module 240 determines that no anomaly is detected in the updated web document.

[0083] Figure 9 900 is a flowchart illustrating example operations of the content publishing platform 102 when performing a method 900 for providing a user-generated content publishing service, according to an example embodiment. The method 900 may be implemented as computer-readable instructions that are executed by one or more processors, such that the operations of the method 900 may be performed in part or in whole by the content publishing platform 102; therefore, the method 900 is described below with reference to an example thereof. However, it should be understood that at least some operations of the method 900 may also be deployed on various other hardware configurations, and the method 900 is not intended to be limited to the content publishing platform 102.

[0084] At operation 905 , anomaly detection system 144 accesses a corpus of revised web documents (eg, stored in database 126 ). Each revised web document included in the corpus includes at least one revision to a portion of a web document published by publishing system 142 .

[0085] At operation 910, the anomaly detection system 144 generates an anomaly score (e.g., text matching score, image similarity score, and numerical deviation score) for each revised web document. Figures 4A to 7 Discuss methods to generate anomaly scores.

[0086] At operation 915 , the improvement module 250 uses the generated anomaly scores to build a decision tree for classifying future revisions. The improvement module 250 also stores the decision tree as a trained model, which is stored in the database 126 .

[0087] At operation 920 , the anomaly detection system 144 receives a revision (eg, an update) to the published web document. The revision may be based on input received from the user 106 via a user interface provided by the content publishing platform 102 and displayed on the client device 110 .

[0088] At operation 925, the anomaly detection system 144 generates one or more anomaly scores for the revised web document. Figures 4A to 7The method discussed generates anomaly scores. At operation 930, one or more anomaly scores are provided to the improvement module 250 for inclusion in the training model.

[0089] At operation 935, the anomaly detection system 144 detects whether an anomaly has occurred in the revised web document based on one or more anomaly scores. At operation 935, if the anomaly detection system 144 does not detect an anomaly in the revised web document, then at operation 940, the anomaly detection system 144 allows the publishing system 142 to publish the revised web document. At operation 940, the anomaly detection system 144 also sends the revised web document to the computer device of the administrator user for manual review. The manual review process allows the administrator user (e.g., a human user) to manually evaluate whether the revised document is abnormal (e.g., includes an anomaly) and provide approval for the revised web document based on the manual evaluation. In other words, if the administrator user determines that the revised web document does not contain an anomaly, he approves the revised web document.

[0090] At operation 945, the content publishing system platform 102 receives the result of the manual review (e.g., whether the administrator user approves the revised web document). If the administrator user approves the revised web document, the content publishing platform 102 notifies the user 106 that the revision is allowed and that the revised web document will be published. For example, the content publishing platform 102 may send a message to the client device 110 to notify the user 106 of this.

[0091] If the administrator user does not approve the revised web document (e.g., because the revised web document is abnormal), the content publishing platform 102 notifies the user 106 that the revision is not allowed and does not publish the revised web document in operation 955. For example, the content publishing platform 102 may send a message to the client device 110 to notify the user 106 of this situation.

[0092] The content publishing platform 102 also provides the results of the manual review to the improvement module 250 for inclusion in the decision tree, and the improvement module 250 further improves the trained model. Depending on the specific results of the manual review, the improvement of the trained model may, for example, include improving one or more threshold anomaly scores, such as a threshold text matching score, a threshold image similarity score, or a numerical deviation score. Improving one or more threshold anomaly scores may include increasing or decreasing the threshold anomaly score.

[0093] At operation 935, if the anomaly detection system 144 detects an anomaly in the revised web document, then at operation 960, the anomaly detection system 144 prevents the publishing system 142 from publishing the revised web document. At operation 960, the anomaly detection system 144 also sends the revised web document to the computer device of the administrator user for manual review. The manual review process allows the administrator user (e.g., a human user) to manually evaluate whether the revised document is anomalous (e.g., includes an anomaly) and provide approval for the revised web document based on the manual evaluation.

[0094] At operation 965, the content publishing platform 102 receives the result of the manual review (e.g., whether the administrator user approves the revised web document 450). If the administrator user does not approve the revised web document (e.g., because the revised web document is abnormal), at operation 955, the content publishing platform 102 notifies the user 106 that the revision is not allowed and does not publish the revised web document. For example, the content publishing platform 102 may send a message to the client device 110 to notify the user 106 of this situation.

[0095] If the administrator user approves the revised web document 450, the content publishing platform 102 provides the results of the manual review to the improvement module 250 for inclusion in the decision tree, and the improvement module 250 further improves the training model. Depending on the specific results of the manual review, the improvement of the training model may, for example, include improving one or more threshold anomaly scores, such as a threshold text matching score, a threshold image similarity score, or a numerical deviation score. Improving one or more threshold anomaly scores may include increasing or decreasing the threshold anomaly score.

[0096] Machine Architecture

[0097] Figure 10 is a block diagram illustrating components of a machine 1000 capable of reading instructions from a machine-readable medium (e.g., a machine-readable storage medium) and performing any one or more of the methodologies discussed herein, according to some example embodiments. Specifically, Figure 10A schematic diagram of a machine 1000 in the example form of a computer system is shown, in which instructions 1016 (e.g., software, programs, applications, applet, app, or other executable code) can be executed to cause the machine 1000 to perform any one or more of the methods discussed herein. For example, the instructions 1016 may include executable code that causes the machine 1000 to perform any one of the methods 500 or 900. These instructions convert a general-purpose, unprogrammed machine into a specific machine that is programmed to perform the functions described and illustrated by the publishing system 142 and the anomaly detection system 144 in the manner described herein. The machine 1000 can operate as a standalone device or can be coupled (e.g., networked) to other machines. In a networked deployment, the machine 1000 can operate in the capacity of a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. As non-limiting examples, the machine 1000 may include or correspond to a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a personal digital assistant (PDA), an entertainment media system, a cellular phone, a smart phone, a mobile device, a wearable device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a network device, a network router, a network switch, a network bridge, or any machine capable of executing, sequentially or otherwise, the instructions 1016 specifying actions to be taken by the machine 1000. Furthermore, while only a single machine 1000 is shown, the term "machine" shall also be taken to include a collection of machines 1000 that individually or jointly execute the instructions 1016 to perform any one or more of the methodologies discussed herein.

[0098] The machine 1000 may include a processor 1010, memory / storage 1030, and I / O components 1050 that may be configured to communicate with each other, for example, via a bus 1002. In an example embodiment, the processor 1010 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), other processors, or any suitable combination thereof) may include, for example, a processor 1012 and a processor 1014 that may execute instructions 1016. The term "processor" is intended to include a multi-core processor 1010 that may include two or more independent processors (sometimes referred to as "cores") that may execute instructions concurrently. Although Figure 10 Multiple processors are shown, but the machine 1000 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0099] The memory / storage device 1030 may include a memory 1032 (e.g., a main memory or other storage device), and a storage unit 1036, both of which are accessible by the processor 1010, for example, via the bus 1002. The storage unit 1036 and the memory 1032 store instructions 1016 that embody any one or more of the methodologies or functionality described herein. During execution of the instructions 1016 by the machine 1000, the instructions 1016 may also reside, completely or partially, within the memory 1032, within the storage unit 1036, within at least one of the processors 1010 (e.g., within a cache memory of the processor), or any suitable combination thereof. Thus, the memory 1032, the storage unit 1036, and the memory of the processor 1010 are examples of machine-readable media.

[0100] As used herein, a "machine-readable medium" refers to a device capable of temporarily or permanently storing or carrying instructions and data, and may include, but is not limited to, random access memory (RAM), read-only memory (ROM), buffer memory, flash memory, optical media, magnetic media, cache memory, other types of memory (e.g., erasable programmable read-only memory (EEPROM)), and / or any suitable combination thereof. The term "machine-readable medium" should be taken to include a single medium or multiple media (e.g., a centralized or distributed database, or associated caches and servers) that can store instructions 1016. The term "machine-readable medium" should also be taken to include any medium or combination of multiple media that can store or carry instructions (e.g., instructions 1016) to be executed by a machine (e.g., machine 1000), such that when the instructions are executed by one or more processors of the machine (e.g., processor 1010), the machine performs any one or more of the methods described herein. Thus, a "machine-readable storage medium" refers to a single storage device or device, as well as a "cloud-based" storage system or storage network comprising multiple storage devices or devices. The term "machine-readable medium" includes machine-readable storage media and transmission media, such as signals.

[0101] I / O components 1050 may include a wide variety of components for receiving input, providing output, generating output, sending information, exchanging information, capturing measurements, etc. The specific I / O components 1050 included in a particular machine will depend on the type of machine. For example, a portable machine such as a mobile phone will likely include a touch input device or other such input mechanism, while a headless server machine will likely not include such a touch input device. It should be understood that I / O components 1050 may include Figure 10Many other components are not shown in the figure. The I / O components 1050 are grouped according to function to simplify the following discussion, and the grouping is not limiting in any way. In various example embodiments, the I / O components 1050 may include output components 1052 and input components 1054. The output components 1052 may include visual components (e.g., displays such as plasma display panels (PDPs), light-emitting diode (LED) displays, liquid crystal displays (LCDs), projectors, or cathode ray tubes (CRTs)), acoustic components (e.g., speakers), tactile components (e.g., vibration motors, resistive mechanisms), other signal generators, etc. The input components 1054 may include alphanumeric input components (e.g., keyboards, touch screens configured to receive alphanumeric input, optical keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touchpads, trackballs, joysticks, motion sensors, or other pointing instruments), tactile input components (e.g., physical buttons, touch screens or other tactile input components that provide location and / or force of touch or touch gestures), audio input components (e.g., microphones), etc.

[0102] In other example embodiments, the I / O component 1050 may include a biometric component 1056, a motion component 1058, an environmental component 1060, or a positioning component 1062, among many other components. For example, the biometric component 1056 may include components for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body postures, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweat, or brain waves), identifying people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or electroencephalogram-based recognition), etc. The motion component 1058 may include an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, a rotation sensor component (e.g., a gyroscope), etc. The environmental component 1060 may include, for example, an illumination sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers that detect ambient temperature), an acoustic sensor component (e.g., one or more microphones that detect background noise), or other components that can provide indications, measurements, or signals corresponding to the surrounding physical environment. The positioning component 1062 may include a position sensor component (e.g., a global positioning system (GPS) receiver component), an altitude sensor component (e.g., an altimeter or a barometer that detects air pressure (altitude can be derived based on the air pressure)), an orientation sensor component (e.g., a magnetometer), etc.

[0103] A variety of technologies can be used to implement communications. The I / O components 1050 may include a communications component 1064 operable to couple the machine 1000 to a network 1080 or device 1070 via coupling 1082 and coupling 1072, respectively. For example, the communications component 1064 may include a network interface component or other suitable device for interfacing with the network 1080. In other examples, the communications component 1064 may include a wired communications component, a wireless communications component, a cellular communications component, a near field communications (NFC) component, a Components (e.g. Low energy consumption), Device 1070 may be another machine or any of a variety of peripheral devices (eg, a peripheral device coupled via a universal serial bus (USB)).

[0104] In addition, the communication component 1064 can detect an identifier or include a component operable to detect an identifier. For example, the communication component 1064 can include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting one-dimensional barcodes (e.g., Universal Product Code (UPC) barcodes), multi-dimensional barcodes (e.g., Quick Response (QR) codes), Aztec codes, Data Matrix, Dataglyph, MaxiCode, PDF417, Supercode, UCC RSS-2D barcodes, and other optical codes), or an acoustic detection component (e.g., a microphone for identifying an audio signal of a tag). In addition, various information can be derived via the communication component 1064, such as location via Internet Protocol (IP), geographic location via Internet Protocol (IP), location information ... Location of signal triangulation, location via detection of NFC beacon signals that can indicate a specific location, and so on.

[0105] transmission medium

[0106] In various example embodiments, one or more portions of network 1080 may be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), the Internet, a portion of the Internet, a portion of a public switched telephone network (PSTN), a plain old telephone service (POTS) network, a cellular telephone network, a wireless network, 1080, or a portion thereof, may include a wireless or cellular network, and coupling 1082 may be a code division multiple access (CDMA) connection, a global system for mobile communications (GSM) connection, or another type of cellular or wireless coupling. In this example, coupling 1082 may implement any of various types of data transmission technologies, such as single carrier radio transmission technology (1xRTT), evolution data optimized (EVDO) technology, general packet radio service (GPRS) technology, enhanced data rates for GSM evolution (EDGE) technology, 3rd Generation Partnership Project (3GPP) including 3G, fourth generation wireless (4G) networks, universal mobile telecommunications system (UMTS), high speed packet access (HSPA), worldwide interoperability for microwave access (WiMAX), long term evolution (LTE) standards, other standards defined by various standards setting organizations, other long range protocols, or other data transmission technologies.

[0107] Instructions 1016 can be sent or received over network 1080 using a transmission medium via a network interface device (e.g., a network interface component included in communication component 1064) and utilizing a plurality of well-known transmission protocols (e.g., Hypertext Transfer Protocol (HTTP)). Similarly, instructions 1016 can be sent or received to device 1070 using a transmission medium via coupling 1072 (e.g., a peer-to-peer coupling). The term "transmission medium" should be taken to include any intangible medium capable of storing, encoding, or carrying instructions 1016 for execution by machine 1000, and includes digital or analog communication signals or other intangible media used to facilitate communication of the software. Transmission media is an embodiment of a machine-readable medium.

[0108] The following numbered examples are embodiments.

[0109] 1. A system comprising:

[0110] one or more processors;

[0111] A computer-readable medium storing instructions that, when executed by one or more processors, cause the system to perform operations comprising:

[0112] accessing a published web document comprising a plurality of different elements generated using data received from a computing device of a user;

[0113] accessing an updated web document based on one or more modifications made by a user to the published web document using an interface presented on a computing device of the user, the updated web document including at least one user-generated modification to an element of the plurality of different elements of the published web document;

[0114] generating one or more anomaly scores based on a comparison of the updated web document to the published web document, the one or more anomaly scores providing a measure of deviation of the updated web document from the published web document; and

[0115] Whether to allow publishing of the updated web document is determined based on a comparison result of the anomaly score and a threshold anomaly score.

[0116] 2. The system of example 1, wherein the plurality of different elements of the published web document include text, images, and numerical values.

[0117] 3. The system of example 1 or example 2, wherein generating the one or more anomaly scores comprises:

[0118] performing a comparison of the updated text included in the updated web document with the published text included in the published web document; and

[0119] generating a text match score based on the comparison, the text match score providing a measure of similarity between the update text and the post text,

[0120] Wherein the one or more anomaly scores include the text matching score.

[0121] 4. The system of example 3, wherein performing the comparison comprises determining similarity between the update text and the published text using a cosine similarity algorithm.

[0122] 5. The system of any of examples 1 to 4, wherein generating the one or more anomaly scores comprises:

[0123] performing a comparison of a published image included in the published web document and an updated image included in the updated web document; and

[0124] generating an image similarity score based on the comparison, the image similarity score providing a measure of similarity between the published image and the updated image,

[0125] Wherein the one or more anomaly scores include the image similarity score.

[0126] 6. The system of example 5, wherein performing the comparison comprises:

[0127] extracting a first set of feature descriptors based on the published image;

[0128] storing a first set of feature descriptors in a first matrix corresponding to the published image;

[0129] extracting a second set of feature descriptors based on the updated image;

[0130] storing a second set of feature descriptors in a second matrix corresponding to the updated image; and

[0131] comparing the first matrix and the second matrix;

[0132] The generating of the image similarity score is based on a comparison of the first matrix and the second matrix.

[0133] 7. The system of any one of examples 1 to 6, wherein:

[0134] Generating the one or more anomaly scores includes generating a value deviation score based on a difference between an updated value included in the updated web document and an average value associated with the web document; and

[0135] The one or more anomaly scores include the numerical deviation score.

[0136] 8. The system of any of examples 1 to 7, wherein generating the one or more anomaly scores comprises:

[0137] generating a text match score based on a comparison of the updated text included in the updated web document and the published text included in the published web document;

[0138] generating an image similarity score based on a comparison of the updated image included in the updated web document and the published image included in the published web page;

[0139] generating a numerical deviation score based on a difference between an updated numerical value included in the updated web document and an average of numerical values associated with the web document; and

[0140] The text matching score, the image similarity score, and the numerical deviation score are aggregated to generate the anomaly score.

[0141] 9. The system of any one of Examples 1 to 8, wherein determining whether to allow publishing of the updated web document based on a comparison of the anomaly score with the threshold anomaly score comprises: in response to the anomaly score crossing the threshold anomaly score, preventing publishing of the updated web document.

[0142] 10. The system according to example 9, further comprising:

[0143] Sending the updated web document to an administrator's device for manual review by the administrator; and

[0144] The one or more threshold anomaly scores are refined based on manual review by an administrator.

[0145] 11. The system of any of examples 1 to 10, further comprising: receiving approval of the updated web document from an administrative computer system as a result of a manual review,

[0146] Wherein improving the threshold anomaly score comprises increasing the threshold anomaly score based on approval of the updated web document as a result of manual review.

[0147] 12. The system of any one of examples 1 to 11, further comprising:

[0148] generating a message indicating that the updated web document is abnormal; and

[0149] The message is sent to a client device of a user responsible for providing the updated web document.

[0150] 13. The system of any one of Examples 1 to 12, wherein determining whether to allow publishing of the updated web document based on a comparison result of the anomaly score and the threshold anomaly score comprises: publishing the updated web document in response to the anomaly score not crossing the threshold anomaly score.

[0151] 14. A method comprising:

[0152] accessing a web document comprising a plurality of different elements generated using data received from a user's computing device;

[0153] accessing an updated web document based on one or more modifications made by a user to the published web document using an interface presented on a computing device of the user, the updated web document including at least one user-generated modification to an element of the plurality of different elements of the published web document;

[0154] generating, using one or more processors, one or more anomaly scores based on a comparison of the updated web document to the published web document, the one or more anomaly scores providing a measure of deviation of the updated web document from the published web document; and

[0155] Whether to allow publishing of the updated web document is determined based on a comparison result of the anomaly score and a threshold anomaly score.

[0156] 15. The method of example 14, wherein generating the one or more anomaly scores comprises:

[0157] generating a text match score based on a comparison of the updated text included in the updated web document and the published text included in the published web document;

[0158] generating an image similarity score based on a comparison of the updated image included in the updated web document and the published image included in the published web page;

[0159] generating a deviation score based on a difference between an updated value included in the updated web document and an average of values associated with the web document; and

[0160] The text matching score, the image similarity score, and the deviation score are aggregated to generate the anomaly score.

[0161] 16. The method of example 14 or example 15, wherein generating the one or more anomaly scores comprises:

[0162] performing a comparison of the updated text included in the updated web document with the published text included in the published web document; and

[0163] generating a text match score based on the comparison, the text match score providing a measure of similarity between the update text and the post text,

[0164] Wherein the one or more anomaly scores include the text matching score.

[0165] 17. The method of any one of examples 14 to 16, wherein generating the one or more anomaly scores comprises:

[0166] performing a comparison of a published image included in the published web document and an updated image included in the updated web document; and

[0167] generating an image similarity score based on the comparison, the image similarity score providing a measure of similarity between the published image and the updated image,

[0168] Wherein the one or more anomaly scores include the image similarity score.

[0169] 18. The method of example 17, wherein performing the comparison comprises:

[0170] extracting a first set of feature descriptors based on the published image;

[0171] storing a first set of feature descriptors in a first matrix corresponding to the published image;

[0172] extracting a second set of feature descriptors based on the updated image;

[0173] storing a second set of feature descriptors in a second matrix corresponding to the updated image; and

[0174] comparing the first matrix and the second matrix;

[0175] The generating of the image similarity score is based on a comparison of the first matrix and the second matrix.

[0176] 19. The method of any one of Examples 14 to 18, wherein:

[0177] Generating the one or more anomaly scores includes generating a value deviation score based on a difference between an updated value included in the updated web document and an average value associated with the web document; and

[0178] The one or more anomaly scores include the numerical deviation score.

[0179] 20. A non-transitory machine-readable storage medium embodying instructions that, when executed by a machine, cause the machine to perform operations comprising:

[0180] publishing a web document comprising a plurality of different elements generated using data received from a user's computing device;

[0181] accessing an updated web document based on one or more modifications made by a user to the published web document using an interface presented on a computing device of the user, the updated web document including at least one user-generated modification to an element of the plurality of different elements of the published web document;

[0182] generating one or more anomaly scores based on a comparison of the updated web document to the published web document, the one or more anomaly scores providing a measure of deviation of the updated web document from the published web document; and

[0183] Whether to allow publishing of the updated web document is determined based on a comparison result of the anomaly score and a threshold anomaly score.

[0184] 21. A machine-readable medium carrying machine-readable instructions that, when executed by at least one processor of a machine, cause the machine to perform the method of any one of Examples 14 to 19.

[0185] Modules, components, and logic

[0186] Certain embodiments are described herein as including logic or multiple components, modules, or mechanisms. A module may constitute a software module (e.g., code embodied on a machine-readable medium or in a transmission signal) or a hardware module. A hardware module is a tangible unit capable of performing certain operations and may be configured or arranged in a particular manner. In an example embodiment, one or more computer systems (e.g., a stand-alone computer system, a client computer system, or a server computer system) or one or more hardware modules (e.g., a processor or a group of processors) of a computer system may be configured by software (e.g., an application or an application portion) as a hardware module that operates to perform certain operations described herein.

[0187] In various embodiments, a hardware module may be implemented mechanically or electronically. For example, a hardware module may include dedicated circuitry or logic that is permanently configured to perform certain operations (e.g., a hardware module may be a dedicated processor, such as a field programmable gate array (FPGA) or an application-specific integrated circuit (ASIC)). A hardware module may also include programmable logic or circuitry that is temporarily configured by software to perform certain operations (e.g., programmable logic or circuitry contained in a general-purpose processor or other programmable processor). It should be understood that the decision to implement a hardware module mechanically, with dedicated and permanently configured circuitry, or with temporarily configured circuitry (e.g., configured by software) may be based on cost and time considerations.

[0188] Thus, the phrase "hardware module" should be understood to encompass a tangible entity that is physically constructed, permanently configured (e.g., hardwired) or temporarily configured (e.g., programmed) to operate in a particular manner or to perform the particular operations described herein. Considering embodiments in which hardware modules are temporarily configured (e.g., programmed), it is not necessary to configure or instantiate each of the hardware modules at any one time. For example, where the hardware modules include a general-purpose processor configured by software, the general-purpose processor can be configured as a respective different hardware module at different times. Thus, software can configure a processor, for example, to constitute a particular hardware module at one time and a different hardware module at another time.

[0189] A hardware module can provide information to other hardware modules and receive information from other hardware modules. Therefore, the described hardware modules can be considered as being communicatively coupled. In the case where a plurality of such hardware modules exist simultaneously, communication can be achieved by signal transmission (e.g., on a suitable circuit and bus connecting the hardware modules). In an embodiment in which a plurality of hardware modules are configured or instantiated at different times, communication between such hardware modules can be achieved, for example, by storing and obtaining information in a memory structure accessible to a plurality of hardware modules. For example, a hardware module can perform an operation and store the output of the operation in a storage device coupled thereto for communication. Another hardware module can then access a memory device later to retrieve and process the stored output. A hardware module can also initiate communication with an input or output device and can operate on a resource (e.g., a collection of information).

[0190] The various operations of the example methods described herein may be performed, at least in part, by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, such processors may constitute processor-implemented modules that operate to perform one or more operations or functions. In some example embodiments, "module" as used herein includes processor-implemented modules.

[0191] Similarly, the methods described herein may be at least partially implemented by a processor. For example, at least some of the operations of the methods may be performed by one or more processors or processor-implemented modules. The execution of certain operations may be distributed among one or more processors and not reside solely in a single machine, but may be deployed across multiple machines. In some example embodiments, one or more processors or processors may be located in a single location (e.g., in a home environment, an office environment, or a server farm), while in other embodiments, the processors may be distributed across multiple locations.

[0192] The one or more processors may also be operable to support execution of related operations in a "cloud computing" environment or as "software as a service" (SaaS). For example, at least some operations may be performed by a group of computers (e.g., machines including processors) that are accessible via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., APIs).

[0193] Electronic devices and systems

[0194] Example embodiments may be implemented in digital electronic circuitry, or in computer hardware, firmware, software, or combinations thereof. Example embodiments may be implemented using a computer program product, such as a computer program tangibly embodied in an information carrier, such as a machine-readable medium executed by or for controlling the operation of data processing apparatus, such as a programmable processor, a computer, or multiple computers.

[0195] The computer program may be written in any form of programming language, including compiled or interpreted languages, and may be deployed in any form, including as a stand-alone program or as a module, subroutine, or other unit suitable for use in a computing environment. The computer program may be configured to execute on one computer or on multiple computers located at one site or on multiple computers distributed across multiple sites and interconnected by a communication network 104.

[0196] In example embodiments, the operations may be performed by one or more programmable processors executing a computer program to perform functions by operating on input data and generating output. The method operations may also be performed by, and the apparatus of example embodiments may be implemented as, special purpose logic circuitry (e.g., an FPGA or ASIC).

[0197] The computing system can include a client and a server. The client and the server are generally remote from each other and typically interact via a communication network 104. The relationship between the client and the server is generated by computer programs that run on respective computers and have a client-server relationship with each other. In embodiments using a programmable computing system, it will be clear that both hardware architecture and software architecture need to be considered. Specifically, it will be clear that implementing a particular function in permanently configured hardware (e.g., an ASIC), in temporarily configured hardware (e.g., a combination of software and a programmable processor), or in a combination of permanently configured and temporarily configured hardware can be a design choice.

[0198] language

[0199] Although embodiments of the present disclosure have been described with reference to specific example embodiments, it will be apparent that various modifications and changes may be made to these embodiments without departing from the broader scope of the subject matter of the present disclosure. Accordingly, the description and drawings should be regarded as illustrative rather than restrictive. The drawings, which form part of specific embodiments in which the subject matter may be implemented, are shown by way of illustration and not limitation. The embodiments shown are described in sufficient detail to enable those skilled in the art to practice the teachings disclosed herein. Other embodiments may be utilized and derived from these embodiments, so that structural and logical substitutions and changes may be made without departing from the scope of the present disclosure. Therefore, this “Detailed Description” section should not be regarded as limiting, and the scope of the various embodiments is limited only by the full scope of the appended claims and their equivalents.

[0200] These embodiments of the subject matter of the present invention are referred to individually and / or collectively by the term "invention" for convenience only and are not intended to actively limit the scope of this application to any single invention or inventive concept (if more than one is actually disclosed). Therefore, although specific embodiments are shown and described herein, it should be understood that any arrangement suitable for achieving the same purpose can be used to replace the specific embodiments shown. This disclosure is intended to cover any and all adaptations or variations of the various embodiments. Combinations of the above embodiments, as well as other embodiments not specifically described herein, will be apparent to those skilled in the art upon study of the foregoing.

[0201] All publications, patents, and patent documents mentioned herein are individually incorporated herein by reference in their entirety. In the event of any inconsistency between the usage herein and the documents incorporated by reference, the usage in the incorporated references should be considered supplementary to that in this document; for irreconcilable inconsistencies, the usage in this document controls.

[0202] In this document, as is common in patent documents, the terms "a" or "an" are used to include one or more rather than one, as distinguished from any other instances or uses of "at least one" or "one or more." In this document, unless otherwise stated, the term "or" is used to refer to a non-exclusive or, for example, "A or B" includes "A but not B," "B but not A," and "A and B." In the appended claims, the terms "including" and "in which" are used as the English-language equivalents of the respective terms "comprising" and "wherein." In addition, in the claims below, the terms "including" and "comprising" are open-ended; that is, a system, apparatus, article, or process that includes matter other than what is listed after the term in a claim will still be considered to fall within the scope of the claim.

Claims

1. A method for detecting anomalies in web document revisions, comprising: accessing a published web document comprising one or more elements generated by a computing device associated with a user; accessing an updated web document generated based on one or more modifications to at least one element of the published web document, the published web document and the updated web document having the same uniform resource identifier (URI); generating one or more anomaly scores based on a comparison of the updated web document with the published web document, the one or more anomaly scores providing a measure of deviation of the updated web document from the published web document; as well as Publishing of the updated web document is blocked based on at least one of the one or more anomaly scores exceeding a corresponding threshold, blocking publishing of the updated web document causing the updated web document to be invisible on one or more computing devices.

2. The method according to claim 1, wherein Generating the one or more anomaly scores includes generating an image similarity score based on a comparison of an updated image in the updated web document and a published image included in the published web document, wherein the published image is associated with a product listing corresponding to a product for sale in an online marketplace.

3. The method according to claim 2, wherein: Generating the one or more anomaly scores further comprises: generating a text match score based on a comparison of update text included in the updated web document and published text included in the published web document, wherein the published text is associated with a product listing corresponding to a product for sale in an online marketplace; generating a deviation score based on a difference between the updated numerical values included in the updated web document and an average of the numerical values associated with the published web documents, wherein the average of the numerical values is associated with a product listing corresponding to a product for sale within an online marketplace; and The text matching scores, image similarity scores, and deviation scores are aggregated to generate the one or more anomaly scores.

4. The method according to claim 1, wherein Generating the one or more anomaly scores includes: performing a comparison of the updated text included in the updated web document with the published text included in the published web document; and generating a text match score based on the comparison, the text match score providing a measure of similarity between the update text and the post text, The one or more anomaly scores include the text matching score.

5. The method according to claim 1, further comprising: performing a comparison of a published image included in the published web document and an updated image included in the updated web document; as well as generating an image similarity score based on the comparison, the image similarity score providing a measure of similarity between the published image and the updated image, Wherein, the one or more anomaly scores include the image similarity score.

6. The method according to claim 5, wherein: Performing the comparison includes: extracting a first set of feature descriptors based on the published image, each feature descriptor of the first set of feature descriptors indicating a corresponding sub-region within the published image; storing a first set of feature descriptors in a first matrix corresponding to the published image, the first matrix comprising an array data structure, the array data structure comprising a set of elements, each element being identified by at least one index or key; extracting a second set of feature descriptors based on the updated image; storing a second set of feature descriptors in a second matrix corresponding to the updated image; and Comparing the first matrix and the second matrix, The generating of the image similarity score is based on a comparison between the first matrix and the second matrix.

7. The method according to claim 1, wherein Generating the one or more anomaly scores includes generating a value deviation score based on a difference between a first value within the published web document and a second value within the updated web document.

8. An anomaly detection system for web document revision, comprising: one or more processors; A non-transitory computer-readable medium storing instructions that, when executed by the one or more processors, cause the system to perform operations comprising: accessing a published web document comprising one or more elements generated by a computing device associated with a user; accessing an updated web document generated based on one or more modifications to at least one element of the published web document, the published web document and the updated web document having the same uniform resource identifier (URI); generating one or more anomaly scores based on a comparison of the updated web document to the published web document, the one or more anomaly scores providing a measure of deviation of the published web document from the updated web document; as well as Based at least in part on at least one of the one or more anomaly scores exceeding a corresponding threshold, determining to block publishing of the updated web document causes the updated web document to be invisible on one or more computing devices.

9. The system according to claim 8, wherein: The one or more elements of the published web document include text, images, and numerical values within a product listing corresponding to the product for sale.

10. The system according to claim 8, wherein: Generating the anomaly score includes: performing a comparison of update text included in the updated web document with published text included in the published web document, wherein the update text is one of the one or more elements; and generating a text match score based on the comparison, the text match score providing a measure of similarity between the update text and the post text, The anomaly score includes a text matching score of the element.

11. The system of claim 10, wherein performing the comparison comprises: The similarity between the update text and the post text is determined using a cosine similarity algorithm that measures a cosine angle between two vectors represented by the update text and the post text.

12. The system according to claim 8, wherein: Generating the anomaly score includes: performing a comparison of a published image included in the published web document and an updated image included in the updated web document, the published image depicting a product for sale in an online marketplace; and generating an image similarity score based on the comparison, the image similarity score providing a measure of similarity between the published image and the updated image, The anomaly score includes the image similarity score.

13. The system according to claim 12, wherein: Performing the comparison includes: extracting a first set of feature descriptors based on the published image; storing a first set of feature descriptors in a first matrix corresponding to the published image; extracting a second set of feature descriptors based on the updated image; storing a second set of feature descriptors in a second matrix corresponding to the updated image; and Comparing the first matrix and the second matrix, The generating of the image similarity score is based on a comparison between the first matrix and the second matrix.

14. The system of claim 8, wherein: Generating one or more anomaly scores includes generating a value deviation score based on a difference between an updated value included in the updated web document and an average of values associated with the published web document, Wherein, the one or more anomaly scores include the numerical deviation score.

15. The system according to claim 8, wherein Generating the anomaly score includes: generating a text match score for a first element of the one or more elements based on a comparison of update text included in the updated web document and published text included in the published web document; generating an image similarity score for a second element of the one or more elements based on a comparison of an updated image included in the updated web document and a published image included in the published web document; generating a numerical deviation score for a third element of the one or more elements based on a difference between an updated numerical value included in the updated web document and an average of numerical values associated with the published web documents; and The text matching score, the image similarity score, and the numerical deviation score are aggregated to generate an aggregate anomaly score.

16. The system of claim 8, further comprising: Transmitting the updated web document to an administrator's device for manual review by the administrator; as well as Based on the results of manual review by administrators, refine one or more threshold anomaly scores.

17. The system of claim 16, further comprising: receiving, from the device, approval of the updated web document as a result of the manual review; as well as The one or more threshold anomaly scores are refined by increasing the one or more threshold anomaly scores based on approval of the updated web document as a result of the manual review.

18. The system of claim 8, further comprising: generating a message indicating that the updated web document is abnormal; as well as The message is transmitted to a client device of a user responsible for providing the updated web document.

19. A non-transitory machine-readable storage medium containing instructions that, when executed by a user's machine, cause the machine to perform operations comprising: generating data on the user's machine associated with one or more elements of the published web document; receiving, on a user's machine, a request to generate an updated web document based on modifying at least one of the one or more elements of the published web document, the published web document and the updated web document having the same uniform resource identifier (URI), wherein the request is transmitted to a system that generates one or more anomaly scores based on a comparison of the updated web document with the published web document, the one or more anomaly scores providing a measure of the deviation of the updated web document from the published web document, and wherein, preventing publication of the updated web document based at least in part on at least one of the one or more anomaly scores exceeding a corresponding threshold; as well as A notification is received on the user's machine indicating that publication of the updated web document is blocked.

20. The non-transitory machine-readable storage medium of claim 19, wherein: The system performs further operations including: generating a decision tree for classification within a training model based on the generation of the one or more anomaly scores; transmitting the updated web document to a computer device associated with an administrative user for review; and The trained model is refined based on the audited analysis results received from the computer device for inclusion in the decision tree, the refinement of the decision tree comprising increasing or decreasing one or more threshold anomaly scores.

Images